efficient backprop yann lecuni, leon bottoui, genevieve b. orr2

44

Upload: hanhi

Post on 13-Feb-2017

223 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

������������ ��������������

��� �!�#"%$'&)(*�,+.-!"/$�01�32406575701(,+'-98:$;�!$�<>=?$�<1$@2:A9BDCECEF6-*�6�*G3H@IJ�6(*KMLON�0 P,$�C75RQTS(*IUI?$�CWVX�Y[Z]\;^;_a`cbed.fW_Wghghikjl^]m�_Wgh_n\�befMo�p4_Wq>\rbhseZ�_Wj.s�tvu�wxuzy9\;{lg�|vm)_Egh_n\�befMo9}*~n�;��� fMo6�l�k�Rp4beik�'_�}

m�_n����\;jl��}1�R�@������� ~E|[�������6}!�a� t��� ik�k�U\�Z�_Eshse_a�4j1ik�'_Ebeghi�s[�.}1���;���6sM\rse_�6shbe_W_7sn}�� \;�k_WZ�}l�4mT���;�;� ~�}>�a�1t���a� p���Y[m4�1u4} m��>� dr��_Eb4 �o>\��lghgh_W_¡.},~r¢r£'¤;����_Ebe�kikj9} � _7beZ]\;j.�

¥ �.\;jlj*} �¦_Ed�j6{!§�¨)be_Wgh_n\�befMo9© \rshsn© fEd�Z�}>^;d;bhb7¨)��ik�k�¦\;Z�_Eshse_�© _n� �9}l� �¦\��lgM¨)ªlbegOsn© ^;Z]�*© �1_d�bei¦^;ikj>\;�k����q1�l{l�kikghol_W��ikj

�)bhbn} � ©>\�j>� �#«�l�k�k_Ebn}�¬@©v­[�4_E�1bM\;�*�4_7s[�®d�be�6gW¯1shbeikfM� g�d�°cseol_4shbM\;�1_W± }� q1beikj1^�_Ebn}*~n�;��¤ ©

²]³c´WµW¶r·>¸;µ�¹ u®o1_�fWd�j6�'_Ebe^;_WjlfE_ºd�°�{>\�f7�.|»q1bed;q>\;^�\�seikd�j¼�k_n\rbejlikjl^#ikg�\;jl\;��� �W_n�ghd�\;g4sed]_E½ q1�U\�ikjºfEd�Z�Z�d�j�q1ol_Wj1d�Z�_Wj1d�j�d;{lgh_Ebe��_n��{6��q1bM\�fEsei�sei¦d;jl_EbegW© � \�j6��1j>�1_EghikbM\�{l�k_]{�_Wo>\n�6ikd;begd;°�{>\�f7�6q bed�q�fn\�j#{�_]\n�'d�i¦�1_W�3��ikseo�shbeikfM�6gseol\�sD\�be_bM\rbe_W���¾_7½1q�d;gh_n�¾ikjºgh_Ebeikd��1g�se_Ef7o1jlikfn\;�%q1�l{l�kikfn\rsei¦d;jlgW©,u®olikgaq>\�q�_Eb�^�ik�'_Egaghd�Z�_d�°cseold�gh_RshbeikfM�6gW}>\�j>��d�¿!_Ebeg�_E½ ql�¦\;jl\�seikd�j1g)d�°���o.�]seol_E�]�®d�be��©� \�j6��\;� seold;begDo>\n�'_�gh�1^�^�_EgOse_n��seo>\�sDgh_WfWd;j>�6|Àd�bM�1_EbDd�q seikZ�i¦�W\�seikd�jÁZ�_Eseo1d6�1g\rbe_®\�� ��\�j.sM\;^�_Ed��lg*°Jd�bcjl_W� bM\;�.j1_Es9shbM\�ikjlikjl^ ©rYÀs�ikg�gho1dn��j�seol\�s,Z�d�gOs)­[fW�¦\;ghghikfn\��k±gh_WfEd�j>�6|»d;bM�1_7bDZ�_7seold6�1gR\�be_�ikZ�q1bM\�fEseikfn\;�c°Jd�b�U\rbe^�_]jl_E�1bM\;�%j1_Es[��d;be�6gW©�t�°J_E�Z�_Eseo1d6�1g�\rbe_aq1bed�q�d;gh_n��seol\�s)�1dDjld;s�ol\r��_4seol_Wgh_a�kikZ�i�sM\�seikd�j1gW©

 ÃlÄ�Å�Æ!ÇaÈDÉÊ,Å!ËeÇ�Ä24�1ÌWÍ>Î!C701Î*�6Ïl�.5E=U01�Á=?K���< $;C7ÐÁÎ,0 Î*(!I?� C]�!$;(!CE� I��!$�5eÑ)01C7ÍÁI?$;� C7�*=U�!Ï#� IUÏ10 CE=¦5EÒ!ÓÔP,$;Ì�� (*K7$=¦5=JKÌ�0 �*Ì�$�Î�5E(*�6I?I?Ð3K7=UÓ�Î!I?$ -,Ìr01Ó�Î*(�5E�657=?0 �*� IUI?к$�Õ�Ì�=U$;�l5;-9� �*G�P,$;Ì�� (*K7$�=¦50 Ö×57$;�ÁÑ)01C7Í�K;AØ�0.Ñ4$�< $;C;-'Ï1$r5M5E=U�*Ï:=U5�570�Ñ40 CEÍ�Ñ4$�I?IÀ-6� �*G�KM01Ó�$�57=?Ó�$;K�570@Ñ40 CEÍÙ�.5�� IUI[- Ì�� ��K7$�$;ÓÚÓ�0 CE$R06Ö�6���6C75�5EÒ*�6�º��K7Ì�=U$;�*Ìr$1A1ÛD$;K7=UÏ1�!=?�!ÏÙ�6�9G�57CW�6=?�!=?�!Ï]�]�!$�5eÑ)01C7Í�(*K7=U�!ÏÙP*� ÌWÍ>Î!CE0 Î�CE$;Ül(!=?C7$�KÓ�� Í>=U�!ÏÓ�� �>Ð@K7$�$;Ó�=U�!Ï1IUÐ@�6CEP!=U57CW�6CEÐDÌWÒ!01=?Ì�$;K�KM(9ÌWÒ]� K/57Ò!$)�l(*ÓÙP,$�C®�6�*G@5eÐ>Î,$;Kv06Ö!�!0�G�$�K�-I?�'Ð1$�CWK�->I?$;�6CE�!=?�!Ï�CE�657$�K�-�5ECE� =U�*=U�!Ï�� �*G¾57$;KM5DK7$r5WK�-*� �*G�KM0�ÖÝ01CM5EÒ%A*ÞRÒ!$�KM$]ÌWÒ*0 =JÌr$;KaÌ;�6��P,$ÌrCE=¦5E=?Ì;�6I[-�Ð $�5/5EÒ!$�CE$�=JK/�!0�ÖÝ0l01IUÎ*C70>06Ö�CE$;Ì�=UÎ,$�ÖÝ01CvG�$�Ìr=JG�=U�*Ïa57Ò!$;ÓßP9$�Ì�� (*KM$�57Ò!$;Ð@� C7$�I?� C7Ï1$�I?ÐÎ!C701P!I?$�Óà� �*G�G!�.5W�:G!$�Î,$��*G�$;�l5;A Ø0.Ñ)$;< $�C�-�57Ò!$;C7$a� C7$RÒ!$;(!C7=JKM57=JÌ�K��6�9G�K70 Ó�$R(!�*G!$�CEIUÐ>=?�!Ï57Ò!$;0 CEÐ�57Ò*�65Ì�� �¾Ò!$;IUÎ�Ï1(!=?G!$@��Î!CW� Ìr57=U57=?0 �!$;CR570�Ó��6Í1$:P,$r5757$;CÌWÒ!0 =JÌr$�K�A

áh�Ù5EÒ!$Râ*CWKe5�K7$;Ìn5E=U01��P9$;IU0.ÑãÑ)$R=?�15EC70�G�(9Ìr$aKe5W�6�*G*�6CWGÙP*�1ÌWÍ>Î!C701Î*�6Ïl�.5E=U01�]�6�*G�G�=JK7Ì�(*K7K���>(!ÓÙP,$�CD06Ö�K7=UÓ�Î!I?$]Ò!$;(!C7=JKM57=JÌ�K�01C�5EC7=JÌWÍ�KaÖÝ0 CD=UÓ�Î!CE0.<>=U�*Ï�=¦5WK�Î,$�C7ÖÝ0 CEÓ�� �*Ìr$1A*äå$Ù�!$�æ>5G�=?KEÌr(9K7K�=JKEKM(!$�K]0 ÖaÌ�0 �>< $;C7Ï1$��*Ì�$ A�äå$�57Ò!$;�çG�$�K7Ì�C7=?P9$��#ÖÝ$�Ñéè7Ì�I?�1K7K7=?Ì;�6IJê¾K7$;Ì�0 �*G�L[01CEG�$;C�!0 ��LOI?=U�!$��6C�01Î�57=?Ó�=Uë��.57=?0 �T5E$;ÌWÒ!�!=JÜl(!$;Kº�6�*GìK7Ò!0.Ñí5EÒ*�.5�5EÒ!$�=?C¾� Î!Î!I?=?Ì;�.5E=U01�ç5E0z�!$�(*CE� I�!$r5eÑ40 CEͼ57CW�6=?�!=?�!ϼ=JK�< $;C7ÐzI?=?Ó�=U57$�Gc-RG�$;K7Î!=U57$#Ó��6�>ÐãÌ�I?� =UÓ�K�5E0¼57Ò*$ÁÌr0 �l5ECE� C7Ðz=?�ì57Ò*$IU=U57$;CE�657(!CE$ A�î�=U�9�6I?IUÐ1-�Ñ)$�Î!CE$;K7$��l5��ÁÖÝ$;ÑïK7$;Ì�0 �*G�L[01CEG�$;CÙÓ�$r5EÒ!0�G!K�57Ò9�.5ºG�0¼� Ì;Ìr$;IU$;CE�657$IU$��6CE�!=U�*Ï�=?��Ìr$�C75E� =U��Ì��1KM$�K�A

ð ñÙò%ó®Æ*ÄË7Ä�ôõó�ÄÈ÷öxò/Ä�ò%Æ*ó®ø7Ëeùcó�Å!ËeÇ�ÄÞRÒ!$�CE$��6CE$�KM$;< $�CW�6I�� Î!Î!CE01� ÌWÒ*$;K570Á�6(!570 Ó��657=JÌ�Ó��1ÌWÒ!=?�!$�IU$��6CE�!=U�*Ï*-cP!(�5�ÓÙ(*ÌWÒú06Ö457Ò*$KM(*Ì;Ìr$�K7KMÖÝ(!I>�6Î!Î!CE01�1ÌWÒ!$;K/Ì��6�ÙP9$4Ì��.5E$�Ï10 CE=Uë;$;G�� K�û6üEý þ.ÿ���������Wý���Wþ ���Wý.ü��9ÿ��lû����������;þ�nA'ÞRÒ*$

Page 2: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

IU$��6CE�!=U�*ÏÁÓ�� ÌWÒ!=?�!$ -®� KÙC7$;Î!CE$;K7$��l57$�Gú=U�çî�=?Ï (!CE$��1-�Ì�0 Ó�Î!(�5E$;K���ÖÝ(!�*Ìr57=?0 ��������� ���ÑaÒ!$�CE$���=JK/5EÒ!$��!L[57Ò�=U�*Î!(�5�Î*�65M57$;C7�/-'� �*G�� C7$;Î!CE$;K7$��l5EK/57Ò!$�Ì�0 I?IU$�Ìn5E=U01�Ù06Ö,� G��e(*KM5E� P!I?$Î*�6CW�6Ó�$r5E$�CWK�=?�ú57Ò!$3KMÐ�KM57$�Ó�A���Ìr0lKe5]ÖÝ(*�*Ìn5E=U01����������� �!�"#������$#���%�r-/Ó�$;�1KM(*C7$�K57Ò!$�G�=JKEÌrCE$�Î*� �*ÌrÐ]P,$r5eÑ4$�$;�!� � -'5EÒ!$�è7Ì�0 CEC7$�Ìn5Wê01C®G!$;K7=UCE$;G�0 (!57Î!(�5�ÖÝ01C�Î*�.5757$;C7�&� � - � �*G57Ò!$:0 (�5EÎ!(�5aÎ!CE0�G�(*Ìr$�G�P>Ð�57Ò!$@K7Ð>KM57$;Ó�AlÞRÒ!$@�'< $;CE� Ï $�Ì�01KM5�ÖÝ(!�*Ìr57=?0 �'��(*) +-,/.0� ����=?K)57Ò*$�'< $�CW�6Ï1$0 Öv5EÒ!$�$�CEC701CEK1����0.< $�CR��K7$r5�06Ö�=U�*Î!(�5-2.0 (�5EÎ!(�5Î9�6=?CEKRÌ�� IUI?$;G¾5EÒ!$@57CW�6=?�!=?�!Ï�KM$�53 ���:+�#�¾+4�46575/5758���:9�;�&9<�-=1A�áh�]5EÒ!$4K7=?Ó�Î*IU$�Ke5�KM$�5M5E=U�!Ï9-�57Ò!$4IU$��6CE�!=?�!ÏÎ!CE0 P*IU$;Ó Ì�0 �*K7=JKe5WK/=?�â*�*G�=?�!Ï�57Ò*$]<.�6I?(!$�06Ö<� 57Ò*�65�Ó�=?�!=?Ó�=Uë;$;K:� (*)#+-,7. � ���rA*áh�#Î!CW� Ìr57=JÌr$1->57Ò!$]Î,$�C7ÖÝ0 CEÓ��6�*Ì�$06Ö�5EÒ!$�KMÐ�KM57$�Ó 0 � �º57CW�6=?�!=?�!Ï�KM$�5@=JK:06Ö�I?=U5M57I?$�=?�15E$�CE$;KM5;AcÞRÒ!$�Ó�0 CE$ÙCE$�I?$�<.�6�l5@Ó�$;� K7(!CE$=?KD57Ò*$�$;C7CE0 CDCE�657$Ù0 Ö�57Ò*$�K7Ð�Ke5E$�Ó =U� 57Ò!$�â*$;I?G%-,ÑaÒ*$�CE$Ù=U5:Ñ40 (!IJGÁP,$�(*KM$�GÁ=?� Î!CW� Ìr57=JÌr$1AÞRÒ!=?K]Î9$;CMÖÝ01C7Ó��6�9Ìr$�=?K]$;KM57=?Ó��657$�G P>ÐÁÓ�$�� K7(!C7=?�!Ï357Ò*$�� Ì�Ì�(!CW� ÌrÐ 0 �ú�#KM$�5]06ÖaK7� Ó�Î*IU$�KG�=?K>�e0 =?�l5)ÖÝCE0 Óõ5EÒ!$:57CW�6=?�!=?�!Ï�K7$r5�->Ì;�6I?IU$�G�5EÒ!$:57$;KM5aKM$�5;A!ÞRÒ!$:Ó�01KM5RÌ�0 Ó�Ó�0 �!I?Ð�(*KM$�GºÌ�01KM5ÖÝ(!�*Ìn5E=U01��=?KR5EÒ!$]Q#$��6�@?�Ül(*�6CE$;GBA�CEC701CDC

� � � �E �>� �GF ���H� � #���;� F ��(*) +-,7.�� �I�J�6Kv+� �

COST FUNCTION

LEARNINGMACHINE

Parameters

Output

E0, E1,....Ep

Error

DesiredOutput

D0, D1,...Dp

Input

M(Z,W)

Z0, Z1,... Zp

W

L<M7N ¹PO>¹ � bM\;�1ik_Wj.sh|»{>\;gh_W���k_n\�bej1i¦j1^@Z]\;fMolikj1_�©

ÞRÒ!=?K�ÌWÒ*�6Î�5E$�C�=JK�ÖÝ0�Ì�(*KM$�G�0 ��KM57CW�.57$;Ï =?$;K�ÖÝ01C�=?Ó�Î*C70.<>=?�!Ï:57Ò!$�Î*C70�Ìr$�K7K�06ÖcÓ�=U�!=?Ó�=Uë;=U�*Ï57Ò!$ºÌ�01KM5�ÖÝ(!�*Ìr57=?0 �%A�Ø�0.Ñ4$�< $;C;-,5EÒ!$;K7$�Ke5ECE�657$;Ï =?$;K@ÓÙ(9Ke5ÙP,$�(*K7$;Gú=?�¼Ìr0 �Q�e(!�*Ìn5E=U01�úÑa=¦5EÒÓ�$�57Ò!0�G!K�ÖÝ0 C�Ó��.æ�=?Ó�=?ë�=?�!Ï357Ò*$��!$r5eÑ40 CEÍSR K@�6P*=UI?=¦5eÐÁ570Áû ��� ��ü7ý �kÿ/T �r-/57Ò*�65]=?K;-c570�Î!CE$;G!=?Ìr557Ò!$�Ìr0 CECE$;Ìn5R5W�6CEÏ $r5WKaÖÝ01C�Î*�.5757$;C7�*K�57Ò!$�IU$��6CE�!=?�!Ï�KMÐ�KM57$;Ó Ò*�1Ka�!0 5DÎ!CE$�<>=?0 (*K7IUÐ3KM$;$���ÖÝ01CÓ�01C7$�G�$�5E� =UIU�nA

Þ/0 (!�*G�$;CEKM5E� �*G¼Ï $;�!$�CW�6I?=?ë;�.5E=U01�%-/I?$r5�(*K�Ìr01�*K7=?G�$;C�Ò*0.Ñ P*� ÌWÍ>Î!CE0 Î*� Ï1�657=?0 �úÑ)01C7Í�K;Aä $#KM5E�6C75�Ña=¦5EÒT�úK7$r5�06Ö@K7� Ó�Î!IU$�K�$;�1ÌWÒç06Ö:ÑaÒ!=JÌWÒã=JK��6�ã=U�!Î*(�5-2.0 (�5EÎ!(�5�Î*� =UC�0 Ö57Ò*$ÖÝ(!�*Ìn5E=U01�º5E0�P,$@IU$��6CE�!$;G%A�?>=?�*Ì�$D5EÒ!$@Ó�$;� K7(!CE$�Ó�$��l54Î!C70�Ì�$;KEK�=JK406Ö×57$;�¾�*0 =JKMÐ1-15EÒ!$�CE$:Ó��'ÐP9$ú$�CECE0 CWK�=?��57Ò*$úKE�6Ó�Î!I?$;K;A4ä $¼Ì�� � =?Ó��6Ï =?�!$ 57Ò*�65�=UÖ�Ñ4$úÌr0 I?I?$;Ìn5E$;GxÓ�(!IU57=?Î!IU$ �����06Ö4KE�6Ó�Î!I?$;K:5EÒ!$��ú$�� ÌWÒúKM$�5@Ñ40 (*I?G IU0>01Í �3IU=U5M5EIU$�G�=/V�$�CE$��l5]P9$�Ì�� (*KM$�0 Ö�57Ò*$��!0 =JKM$�� �*GP9$�Ì��6(9KM$R06Ö,57Ò*$�G�=WV�$�CE$��l5�Î,0 =?�l5EK�KE�6Ó�Î!I?$;GcAXA)� ÌWÒ�06Ö,57Ò!$�KM$�G!�65E�@K7$r5EK®Ñ40 (*I?G��6IJK70DCE$;K7(!IU5

Page 3: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

=U���!$�5eÑ)01C7Í�K�Ña=¦5EÒ�Ó�=?�!=?Ó��:57Ò*�65��6CE$RK7I?=UÏ1Ò15EIUÐ�G�=WV�$�CE$��l5�ÖÝCE0 Ó�$;� ÌWÒ�065EÒ!$�C�� �*GÙÖÝC701ÓÚ57Ò*$57CE(!$�ÖÝ(*�*Ìn5E=U01�%A.áh�]5EÒ!=?K®ÌWÒ*� Î�57$;C;-�Ñ4$4Ì�0 �*Ì�$��l57CW�.5E$�0 ��=UÓ�Î!CE0.<>=U�!Ï57Ò!$4Î!CE0�Ìr$;KEKv06Ö!â*�9G�=U�*Ï57Ò!$RÓ�=U�*=UÓ�(!ÓßÖÝ0 C�5EÒ!$)Î9�6C757=JÌr(!IJ�6C®KM$�5�06Ö*$�æ�� Ó�Î!IU$�Kv57Ò*�65�Ñ4$4� C7$4Ï =?< $;�%A.8:$��!$;CE� IU=?ë;�657=?0 �57$;ÌWÒ*�!=?Ül(!$�K�5EC7Ð35E03Ì�0 CEC7$�Ìn5�ÖÝ01C�57Ò*$�$;C7CE0 CWKa=?�15EC70�G�(9Ìr$;G =U�l5E0�57Ò*$��!$�5eÑ)01C7Í�� KD��CE$;K7(!IU506Ö®0 (!CaÌWÒ!01=?Ì�$:0 Ö®G*�.5E�1KM$�5;A!24065EÒ3� C7$@=?Ó�Î901CM5W�6�l5;A?>$�<1$�CW�6Ia5EÒ!$�01C7$�57=JÌ��6ID$4V�0 C75EK¾Ò*�'< $Á� �*�6I?Ð>ë�$�GT57Ò!$åÎ!CE0>Ì�$;KEK�06Ö]I?$;� C7�*=U�!ÏçPlÐìÓ�=?�!=¦LÓ�=?ë�=?�!Ï�5EÒ!$�$�CECE0 C�0 �¼�35ECE� =U�!=?�!ÏÁK7$r5 �À��Î!CE0>Ì�$;KEK�KM01Ó�$r57=?Ó�$;K�Ì��6I?I?$;G A�Ó�Î!=?C7=JÌ�� I�Na=JKMÍQ#=U�*=UÓ�=?ë;�.5E=U01� � � ��� - � ���OA?>0 Ó�$0 Ö%57Ò*01K7$�57Ò*$�0 CE$r5E=?Ì;�6I��6�*� IUÐ�K7$;K)�6CE$�P*� K7$;G�0 �¾G!$;Ìr01Ó�Î90lKM=?�!Ï�5EÒ!$DÏ1$��!$;CE� IU=?ë;�6L57=?0 ��$�CECE0 C�=?�l570@5eÑ40�57$;C7Ó�KDC P!=J� K)�6�*G�<.�6CE=J�6�*Ì�$ �»K7$�$�$1A Ï9A � � E �*�nAlÞRÒ!$�P!=J� K�=?K)��Ó�$;� K7(!CE$06Ö9Ò*0.ÑzÓÙ(*ÌWÒÙ57Ò!$R�!$�5eÑ)01C7Í@01(�57Î!(!5;-6�'< $;CE� Ï $�G@0.< $�C®�6I?I>Î90lK7K7=?P!IU$RG!�65E�@KM$�5EK®G�=/V,$;CEKvÖÝCE0 Ó57Ò!$G!$;K7=UCE$;GÙÖÝ(!�*Ìr57=?0 �%AlÞRÒ!$�<.�6CE=?� �*Ìr$R=JK��@Ó�$;�1KM(!CE$R06Ö�Ò!0.ÑTÓÙ(*ÌWÒ�57Ò*$��!$r5eÑ40 CEÍ�0 (!57Î!(�5<'� C7=?$;K)P,$r5eÑ4$�$��3G!�.5W� K7$r5WK�A$A�� C7I?Ð�=U�¾57CW�6=?�!=?�!Ï*-157Ò!$@P!=J� K)=JK)IJ�6CEÏ $DP9$�Ì�� (*KM$D57Ò*$D�!$�5eÑ)01C7Í0 (�5EÎ!(�5D=JKÖ»�6C�ÖÝCE0 Óï5EÒ!$�G�$�KM=?CE$;G3ÖÝ(!�*Ìr57=?0 �%A�ÞRÒ!$�<'� C7=J�6�9Ìr$]=JK�<1$�CEÐ3K7Ó��6I?I/P,$;Ì�� (*K7$]57Ò*$G!�.5W��Ò*� KÒ*� G3I?=¦5757I?$Ù=?���*(!$;�*Ìr$]Ð1$r5�A*"/�657$]=?��57CW�6=?�!=?�!Ï*-�5EÒ!$]P!=J� K�=?KDKMÓ��6I?I%P,$;Ì;�6(*K7$�57Ò*$�!$r5eÑ40 CEÍ�Ò*� KDIU$��6CE�!$;GÁ57Ò!$�(!�*G!$�CEIUÐ>=?�!ϺÖÝ(*�*Ìn5E=U01�%A%Ø�0.Ñ4$�<1$�C�-*=UÖ�57CW�6=?�!$�G�570>03IU01�!Ï*-,57Ò*$�!$r5eÑ40 CEÍ�Ña=?I?I�� I?K70]Ò*�'<1$�I?$;�6CE�!$�G�57Ò!$:�!01=?K7$DK7Î9$�Ìr=Uâ9Ìa5E0Ù5EÒ*�.5RG!�.5W� K7$r5�AlÞRÒ!=JK4=?K)CE$rÖÝ$�CECE$;G570#� K:0.< $;CM5ECE� =U�*=U�!Ï9A*áh�¼K7(*ÌWÒ ��Ì��1KM$1-,57Ò!$�<.�6CE=?� �*Ìr$�Ña=?IUI�P,$�I?� C7Ï1$�P,$;Ì;�6(*K7$Ù5EÒ!$��!01=?K7$<'� C7=?$;K�P,$r5eÑ4$�$;��G!�65E�1KM$�5EK;A�áO5�Ì;�6�ÙP,$aKMÒ*0.Ña�]57Ò9�.5�57Ò*$4Ó�=?�!=?ÓÙ(!Óß5E065W�6I�$�CECE0 C�Ña=UI?I�0�Ì�Ìr(*CÑaÒ!$��357Ò*$]KM(!Ó 06Ö�P!=?�1Ka�6�*G3<.�6CE=J�6�*Ì�$:� C7$@Ó�=U�*=UÓ��6I[A

ÞRÒ!$�CE$3� C7$#�å�>(!ÓÙP,$�C�0 Ö5E$;ÌWÒ!�!=JÜl(!$;K��Ý$ A Ï*A�$��6CEIUÐzKM5701Î!Î!=?�!Ï*-�CE$�Ï1(!IJ�6CE=Uë��.57=?0 � ��ÖÝ01CÓ��6æ�=UÓ�=?ë�=?�!Ï�57Ò!$�Ï1$��!$;CE� IU=?ë;�657=?0 �¾� P!=?IU=U5eÐ�06Ö����!$r5eÑ40 CEÍ�ÑaÒ!$;�3(9KM=?�!Ï�P*� ÌWÍ>Î!CE0 Î/A

ÞRÒ!$ú=?G�$��ç0 ÖÙ5EÒ!=?K#ÌWÒ9�6Î�5E$�C�-)5EÒ!$�CE$rÖÝ01C7$1-R=?K35E0TÎ*C7$�KM$;�15�Ó�=U�*=UÓ�=?ë;�.5E=U01� KM57CW�.5E$�Ï1=U$�K�ÝÏ =?< $;�Á�ºÌ�01KM5�ÖÝ(!�*Ìr57=?0 � �D�6�*G35EÒ!$]5EC7=JÌWÍ�K�1K7K70�Ìr=J�.57$�G�Ña=¦5EÒÁ=?�*ÌrCE$;�1KM=?�!Ï�57Ò*$�K7Î9$;$;GÁ� �*GÜ1(9�6I?=¦5eÐz0 Ö5EÒ!$�Ó�=?�!=?Ó�=Uë��.57=?0 �/A�áO5�=?K�Ò!0.Ñ4$�<1$�C�ÌrI?$;�6C�5EÒ*�.5�5EÒ!$#ÌWÒ!0 =JÌr$306ÖD57Ò!$#Ó�0�G�$�I�ÝÓ�0�G�$�I%K7$�I?$;Ìr57=?0 � �r-15EÒ!$��6CWÌWÒ!=U57$;Ìr57(!CE$D� �*Gº57Ò!$�Ìr0lKe54ÖÝ(!�*Ìr57=?0 �3=?KRÌ�C7(*Ì�=?� I,ÖÝ0 CR0 P�5W�6=?�!=?�!Ï���!$r5eÑ40 CEÍ]5EÒ*�.5)Ï $��*$�CW�6I?=Uë;$;K�Ñ4$�I?IÀA�?>0]Í1$�$�Î�=?��Ó�=U�9G�57Ò*�65)=UÖc57Ò!$ÑaC701�!Ï�Ó�0>G!$�I�ÌrIJ� KEK�=?K(*KM$�GÙ� �*G��!0:Î!CE0 Î,$�C�Ó�0�G�$;I�KM$;IU$�Ìn57=?0 �Ù=?K®G�01�!$ -;57Ò!$;��$;< $;�Ù�K7(!Î,$�CEPÙÓ�=U�*=UÓ�=?ë;�.5E=U01�ÙÑa=?IUIÌrI?$;�6CEI?Ð��!065RÒ!$�I?Î�< $;C7Ð�ÓÙ(9ÌWÒ%A�áh�ºÖ»�1Ìn5�->57Ò!$@$ræ�=JKe5E$��*Ì�$@06Ö/0.<1$�C757CW�6=?�!=U�*ÏÙÒ*�1K)I?$;G3K7$�< $;CE� I�6(�5EÒ!0 CWKR570¾KM(!Ï1Ï $�Ke5a5EÒ*�.5=?�*�1Ì�Ìr(*CE�657$�Ó�=U�*=UÓ�=?ë;�.5E=U01� �6I?Ï 01C7=U57Ò*Ó�KaÌ;�6�#P,$]P,$r5M5E$�CD57Ò*� �Ï 0>0�Gº01�!$;K;A

4Å>ó�ÄDÈ�ó®Æ*È��úó�Ê� ��DÆ!Ç��ó®ô�ó�Å!ËeÇ)Ä

��IU57Ò!01(!Ï Òú5EÒ!$�5EC7=JÌWÍ�KÙ�6�9G¼�6�*� IUÐ�K7$;K�=?�ú57Ò!=JK�Î*�6Î,$�C��6CE$�Î!C7=?Ó��6CE=UI?Ð Î*C7$�KM$;�15E$;Gú=?�¼57Ò*$Ìr0 �l5E$ræ>5Ù06Ö�èEÌrIJ� KEKM=JÌ�� I?ê3ÓÙ(*I¦5E=¦LOI?�'Ð1$�C�ÖÝ$;$;G>L[ÖÝ0 CEÑR�6CWG �*$�(!CW�6I��*$r5eÑ40 CEÍ>K;-%Ó��6�>Ðå06ÖR57Ò*$�Ó�6IJKM0��6Î*Î!IUÐ�570�Ó�01KM5�0657Ò*$�CaÏ CW� G!=U$;�157L[P9� K7$;G�I?$;� C7�!=?�!Ï�Ó�$r5EÒ!0�G!K�A

ÞRÒ!$)K7=UÓ�Î!I?$;KM5/ÖÝ01C7Ó 0 Ö!ÓÙ(!IU57=?I?�'Ð1$�C%I?$;� C7�*=U�!ÏÓ��1ÌWÒ!=?�!$�5ECE� =U�!$�G�Ña=U57Ò]Ï1CE�1G�=U$;�l5MLOP*� K7$;GIU$��6CE�!=U�*ÏD=JK�K7=UÓ�Î!I?ÐÙ�@Ke5W� ÌWÍ]0 Ö9Ó�0�G�(!I?$;K;-6$;�1ÌWÒ�0 Ö9ÑaÒ!=JÌWÒ�=UÓ�Î!I?$�Ó�$��l5WK��DÖÝ(!�*Ìn5E=U01����. �� . ��� . �� .�� + �n-lÑaÒ!$�CE$�� . =?K)�]<1$;Ìn5E0 C�CE$�Î*C7$�KM$;�15E=U�*Ï@57Ò!$D0 (�5EÎ!(�5)0 Ö�57Ò!$DÓ�0�G�(*IU$1-�� .=?K)57Ò*$:<1$;Ìr570 C406Ö/57(!�*� P!I?$:Î9�6CW�6Ó�$r5E$�CWK�=?��5EÒ!$@Ó�0�G�(*IU$ �»��K7(!P*K7$r5R06Ö����r-��6�*G�� .�� + =?K57Ò!$:Ó�0>G!(!IU$ R K)=?�!Î!(�5R< $�Ìn5701C �À� K)Ñ)$;IUIc� K�57Ò!$:Î!CE$�<>=?0 (*K�Ó�0�G�(*IU$ R K40 (!57Î!(�5R< $�Ìn5E0 C-�nA1ÞRÒ*$=U�!Î*(�5����:5E0Ù5EÒ!$:â*CEKM5RÓ�0�G�(*IU$@=JK�5EÒ!$@=U�*Î!(�5aÎ*�.5757$;C7�@�1��A>áOÖ/57Ò*$:Î9�6C757=J�6IcG�$�CE=?<'�657=?< $D06Ö���ÙÑa=¦5EÒºCE$;K7Î9$�Ìn54570�� . =?K4Í>�!0.Ña�%-157Ò*$��¾57Ò!$:Î*�6C757=J�6I�G�$;C7=?<.�.57=?< $�K�0 Ö ���ÙÑa=U57Ò¾C7$�KMÎ,$;Ìr5

Page 4: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

570 � .¾� �*G � .�� + Ì�� �¾P,$]Ìr01Ó�Î*(�57$�G¾(9KM=?�!Ï�57Ò!$�P9� ÌWÍ>Ñ4� CEGºC7$�Ìr(!CECE$��*Ì�$� � �� � . � � �

� � � � . �� .�� + � � � �� � .

� ���� � .�� + � � �

� � � � . � .�� + � � �G�� ��. � ���

ÑaÒ!$�CE$�������

� � . ��.�� + �a=?K�57Ò!$�l� Ì�0 P!=J�6��0 Ö � Ña=U57Ò#CE$;K7Î9$�Ìn5�5E0 � $�<.�6I?(*�657$;G#�65�57Ò*$Î901=U�l5!����. �� .�� + �n-��6�*G���

���� � . ��.�� + ��=?K57Ò*$��l� Ìr01P!=J�6��06Ö � Ña=U57Ò CE$;K7Î,$;Ìn5570 �çA

ÞRÒ!$ �1�1Ìr01P!=?� �º0 Ö®��<1$;Ìn5E0 C4ÖÝ(!�*Ìr57=?0 �#=JKa��Ó��.57CE=Uæ¾Ìr01�15W�6=?�!=?�!Ï�57Ò!$�Î*� CM5E=?� I%G�$�CE=?<'�657=?< $�K06Ö]�6I?I�5EÒ!$Á0 (!57Î!(�5WKºÑa=U57ÒxCE$;K7Î,$;Ìn5º570ç� IUI57Ò!$Á=?�!Î!(!5EK;A)äxÒ!$;�x57Ò*$ �6P,0.< $#$�Ü1(9�.57=?0 �9K�6CE$3� Î!Î!I?=U$�Gz5E0 5EÒ!$#Ó�0�G�(!I?$;K�=?�ãCE$�<1$�CWKM$¾0 CWG�$�C�-�ÖÝC701Ó I?�'Ð1$�C�� 5E0úI?�'Ð1$�CB� -��6I?IR57Ò*$Î*�6C757=J�6I,G�$�CE=U<.�657=?< $;K�06Öc5EÒ!$DÌ�01KM5�ÖÝ(!�*Ìr57=?0 �¾Ña=¦5EÒ�C7$�KMÎ,$;Ìr5�5E0��6I?I!5EÒ!$:Î*�6CW�6Ó�$r5E$�CWK�Ì�� ��P,$Ìr0 Ó�Î!(!57$;G%A*ÞRÒ!$�ÑR�'Ð�06Ö�Ì�0 Ó�Î!(�5E=U�*Ï�Ï CW� G�=?$��l5EK4=JKaÍ>�!0.Ña��� K4P*� ÌWÍlLOÎ!CE0 Î*� Ï1�.5E=U01�%A

Þ/CW� G�=U57=?0 �9�6IlÓÙ(!IU57=UL[IJ�'Ð $;C/�!$;(!CW�6I>�!$r5eÑ40 CEÍ�K/� C7$4�DK7Î9$�Ìr=J�6I�Ì��1KM$)06Ö!5EÒ!$R�6P,0.< $)K7Ð�Ke5E$�ÓÑaÒ!$�CE$:57Ò!$�Ó�0�G�(!I?$;Ka� C7$��6IU57$;C7�9�.57$�G�IJ�'Ð $;CEK406Ö�Ó��.57CE=UæºÓÙ(!IU57=?Î!I?=?Ì;�.5E=U01�*K �×5EÒ!$�Ñ4$�=?Ï Òl5EK ��6�*G3Ìr01Ó�Î,0 �*$��l5MLOÑa=?K7$�KM=?Ï Ó�01=?GºÖÝ(!�*Ìr57=?0 �*K �Ý57Ò!$�(*�!=¦5WK#�4C

� . � � . � .�� + � E �� .�� � ��� . � ��� �

ÑaÒ!$�CE$ ��.¾=JK���Ó��.57CE=UæºÑaÒ!01K7$@�>(!ÓÙP,$�C�0 Ö®Ì�0 I?(!Ó��*Ka=JKR57Ò*$]G�=UÓ�$;�*KM=?0 ��0 Ö � .�� + -*� �*G�l(*ÓÙP,$�CÙ0 ÖaC70.Ñ�K@=JK�5EÒ!$¾G�=?Ó�$��*K7=?0 �¼06Ö�� .,A �ï=JKÙ��<1$;Ìn5E0 C�ÖÝ(*�*Ìn5E=U01�ú57Ò*�65��6Î*Î!IU=?$;KÙ�KM=?Ï Ó�0 =JG ÖÝ(!�*Ìr57=?0 �ú570#$�� ÌWÒúÌr01Ó�Î,0 �*$��l5]06Öa=U5EK�=?�!Î!(�5�A�� . =?K]57Ò!$�<1$;Ìn5E0 C�0 ÖRÑ4$�=?Ï Òl5E$;GKM(!Ó�K;-!0 C � ���hý��%ÿ������ � n-�570�IJ�'Ð $�C���A��Î!Î!I?Ð>=U�*Ï�57Ò!$DÌWÒ*�6=?��C7(*IU$a5E0�57Ò*$�$;Ül(*�657=?0 ���6P,0.< $1-.57Ò*$ÌrIJ� KEKM=JÌ�� I!P*�1ÌWÍlÎ*C701Î*�6Ïl�.57=?0 �$;Ül(*�.5E=U01�*Ka�6CE$:0 P�5W�6=?�!$;G C

� ������ ,. ������� � ,. � � �G�

��� ,. � � �� �G���� ,! . � � .�� + � ���

��� ,. �#" �� �������$.�� + � J , � , $. � ���

�%� ,. 5 ��& �ÞRÒ!$]�6P,0.< $:$;Ül(*�657=?0 �*KaÌ;�6���6IJKM0�P9$�ÑaCE=U5M57$;��=U��Ó��.5EC7=Uæ�ÖÝ0 CEÓ'C

� ���� � . � �'� �#� . � � ���

� � . �)(X�� ���� � . � � .�� + � �G�

� � . ��* �� �G�� � .�� + � �,+. � ���

� � . 5 ��- �ÞRÒ!$�KM=?Ó�Î!IU$�Ke5aI?$;� C7�!=?�!Ï'�ÝÓ�=?�!=UÓ�=?ë;�657=?0 � �4Î!CE0�Ìr$�G�(!CE$:=?��KM(*ÌWÒ���KM$�5M5E=U�!Ï�=JK457Ò!$�Ï1CE�6L

G�=U$;�l5G�$;KEÌr$;�15�� IUÏ10 CE=¦5EÒ!Ó�ÑaÒ!$;C7$ � =JKa=U57$;CE�657=?< $�I?Ð�� GP�e(*Ke5E$;G�� K4ÖÝ01IUI?0.Ñ�KDC���/.%� � ���/. F � � F10 � �

� � 5 �%� � �

Page 5: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

áh� 5EÒ!$�KM=?Ó�Î!IU$�Ke5�Ì��1KM$1- 0 =?K@�3K7Ì;�6IJ�6C@Ìr01�*Ke5W�6�l5;A�Q#01C7$�KM01Î!Ò!=JKe5E=?Ì;�.5E$;GÁÎ*C70�Ìr$�G�(!CE$;KD(*K7$<'� C7=J�6P*IU$ 0 A�áh�T065EÒ!$�C�Ó�$r5EÒ!0�G!K 0 5W�6Í $�K]5EÒ!$3ÖÝ0 CEÓ 06Ö@� G�=J�6Ï10 �*� I)Ó��657CE=¦æc-®0 C�=JK�� �$;KM57=?Ó��657$¾06ÖR57Ò!$¾=U�><1$�CWKM$�Ø$;KEKM=J�6�¼Ó��.5EC7=Uæú06Öa57Ò*$ºÌr0lKe5�ÖÝ(!�*Ìn5E=U01� �»K7$;Ì�0 �*G¼G!$�CE=U<.�.5E=U<1$Ó��657CE=¦æ ��K7(*ÌWÒz�1K]=?�¼57Ò!$���$�ÑR5E0 �ç� �*G��D(*�1KM=UL��$�ÑR5E0 �úÓ�$r57Ò*0>G*K�G�$�K7Ì�C7=?P,$;GúIJ�.57$;CÙ=?�57Ò!$�ÌWÒ*� Î�57$;C;AS� Î!C701Î9$;C�ÌWÒ!0 =JÌr$�06Ö 0 =JK@=?Ó�Î,0 C75E� �l5@� �*GÁÑa=?IUI�P,$�G�=JK7Ì�(*K7K7$;G �.5@IU$;�!Ï65EÒI?�657$�C�A

� ���aò�� Æ!ó�Ê,Å!ËMÊ�ó�ø� �Æ9ËMÊ� ��

24�1ÌWÍ>Î!C701Î*�6Ïl�.5E=U01��Ì��6�3P,$�< $�CEÐ�K7IU0.Ñ�Î9�6C757=JÌr(!IJ�6CEIUÐ�ÖÝ01CaÓÙ(!IU57=?IJ�'Ð $�CE$;Gº�!$r5eÑ40 CEÍ�K4ÑaÒ!$�CE$57Ò!$#Ìr0lKe5�KM(*CMÖ»�1Ìr$3=?K�5eÐ>Î!=JÌ��6I?I?Ðå�*0 ��LhÜl(*� G�CW�.5E=?Ì -v�*0 ��LhÌr01�l<1$ræc-v� �*GzÒ*=UÏ1ÒTG�=?Ó�$��*K7=?0 �*� IÑa=¦5EÒ Ó��6�>Ð�I?0>Ì;�6I®Ó�=U�*=UÓ��3�6�*G 2.0 C �9�.5@CE$�Ï =?0 �9K�A�ÞRÒ!$�CE$�=?K:�!0¾ÖÝ0 CEÓÙ(!IJ��5E0¾Ï1(*�6CW�6�l5E$�$57Ò*�65:� � �/5EÒ!$a�!$r5eÑ40 CEÍ�Ña=?IUI*Ì�0 �>< $;C7Ï1$�5E0@�:Ï10l0�G�KM01IU(!57=?0 �%-"� E �®Ìr01�>< $�CEÏ $;�*Ìr$)=?K�K7Ña=UÖ×5;-601C��� ��Ìr01�l<1$�CEÏ $;�*Ìr$a$;< $���0�Ì;Ìr(!CWK��654�6I?I[AlØ0.Ñ)$;< $�C�-6=?��57Ò*=?K4KM$�Ìn5E=U01��Ñ)$DG�=?KEÌr(9K7K4���>(!ÓÙP,$�C06Ö�5EC7=JÌWÍ�K�5EÒ*�.5�Ì�� �ÁÏ CE$;�657I?о=?Ó�Î!C70.<1$]5EÒ!$�ÌWÒ9�6�*Ì�$;KD06Ö�â*�*G!=U�!Ï���Ï10>0>G KM01IU(!57=?0 �ÁÑaÒ!=?I?$�6IJKM0G!$;ÌrCE$;�1KM=?�!Ïa5EÒ!$)Ì�0 �>< $;C7Ï1$��*Ì�$®57=?Ó�$)0 Ö×57$��ÙPlÐ@01CEG�$;CEK%06Ö*Ó��6Ï1�!=U57(*G�$1A�QÁ0 CE$�G�$r5W�6=?IU$�G57Ò!$;0 CE$r57=JÌ�� I��e(*KM57=Uâ9Ì��657=?0 �*KRÑa=?IUI/P9$�Ï1=U<1$��3=U��IJ�.57$;C�KM$�Ìn5E=U01�*K�A

����� �����������! ���"��$#&%(') �*� ,+��!���-�/.0%1�2')3�"03�4��

�45º$;�1ÌWÒx=¦5E$�CW�.57=?0 �/-�$;Ül(*�657=?0 ��� � � ��CE$;Ül(!=?C7$�K��zÌ�0 Ó�Î!I?$r57$#Î9� KEK�57Ò*C701(!Ï ÒT57Ò*$Á$��l57=?CE$G!�.5W� K7$r5�=?�#0 CWG�$�Ca5E0ºÌr01Ó�Î!(�57$]57Ò!$¾ý65���ü7ý;û �0 Ca5EC7(*$]Ï CW� G�=?$��l5�A*ÞRÒ!=JK�=JKaC7$�ÖÝ$�CEC7$�G¾570º�1KP*�.5WÌWÒ�I?$;� C7�*=U�!Ï�K7=U�9Ìr$��6��$��l5E=UCE$�èMP*�65EÌWÒ*ê:0 Ö�G!�65E�:ÓÙ(9Ke5�P,$�Ìr01�*K7=?G�$;C7$�GÙP,$rÖÝ01C7$aÑ4$�=?Ï Òl5WK�6CE$ (!Î�G!�.5E$;GcA �I¦5E$�CE�*�.5E=U<1$�I?Ð -R0 �*$ Ì�� � (*K7$úKe5E0>ÌWÒ9� KM57=JÌ��Ý01�!I?=U�!$��¾IU$��6CE�!=U�*ÏçÑaÒ!$;C7$å�Wÿ��>û � �a$ræ!� Ó�Î*IU$ 3 � ( ;� ( =�=JK�ÌWÒ!01K7$�� �Ý$1A Ï9A!CW�6�*G�01Ó�IUÐ"�)ÖÝCE0 Ó 5EÒ!$�57CW�6=?�!=?�!Ï�KM$�5�.5�$�� ÌWÒ=¦5E$�CW�.57=?0 ��.nAX�� ����[ÿ���ý � ��06Ö,57Ò!$�57CE(!$aÏ CW� G�=?$��l5�=JK®57Ò!$;��Ìr01Ó�Î*(�57$�G�P9� K7$;G�0 ��57Ò*$�$�CEC701C� ( 0 Öv5EÒ*�.5a$ræ!� Ó�Î*IU$1-*�6�*Gº57Ò*$��357Ò!$�Ñ4$�=?Ï Òl5WKa�6CE$D(!Î�G!�657$;G C

����.�7 � �<� ���/.%� F10 � � (� � 5 �%�X���

2)$�Ì�� (*KM$�57Ò!=JK/$�Ke5E=UÓ��657$�0 Ö�57Ò!$)Ï CW� G�=?$��l5/=JK%�*0 =JKMÐ1-�5EÒ!$�Ñ4$�=?Ï Òl5EK/Ó��'ÐD�!0 5vÓ�0.< $�Î!C7$�Ìr=JKM$;IUÐG�0.Ña��57Ò*$)Ï1CE�1G�=U$;�l5v�.5�$;� ÌWÒÙ=¦5E$�CW�.57=?0 �/A �DK/Ñ4$)K7Ò*�6I?I>KM$;$ -;57Ò!=JK:èM�!01=?K7$;ê��65v$�� ÌWÒ]=U57$;CE�657=?0 �Ì��6� P9$�� G�<.� �15W�6Ï1$�0 (9K�AS?l570�ÌWÒ*�1Ke5E=?ÌÙIU$��6CE�!=U�*Ϻ=JKÏ1$��!$;CE� IUI?Ð357Ò!$�Î!CE$rÖÝ$�CECE$;G�Ó�$r5EÒ!0�GÁÖÝ01CP*� K7=?Ì:P*�1ÌWÍ>Î!C701Î*�6Ïl�.5E=U01��ÖÝ0 CR5EÒ!$@ÖÝ0 I?IU0.Ña=?�!Ï�57Ò!CE$�$�CE$;�1KM01�*KDC

8�9 #!�:3;���!4<%1 =�&>?�����������! ���"��A@B%1�2')3C"�3C4� A ?l5E0>ÌWÒ9� KM57=JÌ�I?$;�6CE�!=?�!Ï�=?K4(*K7(*�6I?I?Ð ���:D �@Ö»�1Ke5E$�C457Ò9�6�3P*�.5WÌWÒºI?$;�6CE�!=?�!Ï*AE A ?l5E0>ÌWÒ9� KM57=JÌDI?$;� C7�*=U�!Ï��6IJK70�0 Ö×57$;�3CE$;K7(!IU5EKR=?�¾P,$r5757$;CKM01IU(!57=?0 �*K;A�!A ?l5E0>ÌWÒ9� KM57=JÌDI?$;� C7�*=U�!Ï�Ì�� �¾P,$�(*K7$;G¾ÖÝ0 CR5ECE�1ÌWÍ>=U�!Ï�ÌWÒ*� �!Ï $�K�A

?l570�ÌWÒ*�1Ke5E=?Ì�IU$��6CE�!=U�*Ï=JK/Ó�01KM5®06Ö×57$;� � �:D �aÖ»� KM57$;C/5EÒ*�6��P*�.5WÌWÒÙI?$;� C7�!=?�!Ï�Î9�6C757=JÌr(!IJ�6CEIUÐ0 ��IJ�6CEÏ $aCE$;G�(*�*G!�6�l5)G*�.5E�1KM$�5EK;AlÞRÒ*$CE$;� K70 ��ÖÝ01C�57Ò*=?K)=?K4K7=UÓ�Î!I?$�5E0�K7Ò!0.Ñ�A>&)01�*KM=JG�$;C�57Ò*$KM=?Ó�Î!IU$3Ì��1KM$¾ÑaÒ!$�CE$¾��5ECE� =U�*=U�!ÏåKM$�5�0 ÖKM=?ë�$ � � ��� =JKÙ=?�*� G!< $�C757$;�l57I?ÐåÌ�0 Ó�Î90lKM$�G¼06Ö � �=?G�$;�l57=JÌ��6IvÌr0 Î*=U$�Ka06Ö���K7$r5Ña=U57Ò � � � K7� Ó�Î!IU$�K�A ��< $;CE� Ï =?�!Ï�57Ò!$]Ï1CE�1G�=?$��l5�0.< $;C��6I?I<� ��� �Î*�.5757$�CE�*KÙÏ =?< $;K]57Ò!$¾$ræ!� Ìr5�KE�6Ó�$�CE$;K7(!I¦5�� K�Ìr0 Ó�Î!(!57=?�!Ï#57Ò*$ºÏ CW� G!=U$;�15�P*� K7$;Gú01�'�e(*Ke557Ò!$3â*CWKe5@� � � A�ÞRÒ>(*K;-�P*�.5WÌWÒçÏ1CE�1G�=U$;�l5�G�$�K7Ì�$��l5�=?K�ÑR� KM57$�ÖÝ(!IaP,$;Ì�� (*K7$¾=U5�C7$�Ìr01Ó�Î*(�57$�K

Page 6: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

57Ò!$�KE�6Ó�$�Ül(*�6�l5E=¦5eÐ � � 57=?Ó�$�K�P,$rÖÝ0 CE$�0 �*$�Î*�6CW�6Ó�$r5E$�C@(!Î�G!�.5E$ A�BD� 5EÒ!$�065EÒ!$�C�Ò9�6�*Gc-Ke5E0>ÌWÒ9� KM57=JÌÁÏ CW� G!=U$;�153Ña=UI?I�KM$;$å�zÖÝ(!I?I�$�Î,0>ÌWÒ � K � � =¦5E$�CW�.57=?0 �9K�5EÒ!C701(!Ï Ò�� � � � LOI?0 �!Ï57CW�6=?�!=U�*Ï�K7$r5�A%áh�úÎ!CW� Ìn5E=?Ì�$ -c$�æ�� Ó�Î!IU$�K@CE� C7$;IUÐÁ� Î!Î,$;�6C@Ó�0 CE$�57Ò*� �ú0 �*Ì�$�=?�¼�#G!�.5W� K7$r5�-P!(�5:57Ò!$;C7$Ù� C7$Ù(*KM(9�6I?IUÐ#ÌrI?(*KM57$;CEK06Ö�Î*�65M57$;C7�9K�5EÒ*�.5@�6CE$]< $;C7Ð�KM=?Ó�=UIJ�6C�A9î*0 CD$ræ!�6Ó�Î!I?$Ù=?�Î!Ò!0 �*$�Ó�$�ÌrIJ� KEKM=Uâ9Ì��657=?0 �%-�� IUIc0 Ö/57Ò*$:Î9�.5M5E$�CE�*K4ÖÝ0 CR5EÒ!$@Î!Ò!01�!$�Ó�$ 2��:2ÙÑa=?IUI<�»Ò!0 Î,$rÖÝ(!I?I?Ð$�Ìr0 �l5W�6=?�]ÓÙ(*ÌWÒ�06Ö!5EÒ!$RK7� Ó�$4=?��ÖÝ0 CEÓ��.57=?0 �/A;áO5�=JK/5EÒ!=JKvCE$;G!(!�*G!� �*ÌrÐ:57Ò9�.5�Ì��6��Ó��6Í $)P*�.5WÌWÒIU$��6CE�!=U�*Ï�Ó�(*ÌWÒ�KMI?0.Ñ)$;C457Ò*� �¾01��L[I?=?�!$ A?l570�ÌWÒ*�1Ke5E=?ÌIU$��6CE�!=U�*Ï�� I?K70Ù0 Ö×57$��¾C7$�KM(*I¦5WK)=?�3P9$�5M57$;CaKM01IU(�5E=U01�*K4P9$�Ì�� (*KM$:06Ö/57Ò!$@�!01=?K7$=U��5EÒ!$:(!Î,G*�.57$�K�A1��01�!I?=U�!$��6C)�!$r5eÑ40 CEÍ�K�(*K7(*�6I?I?Ð�Ò*�'<1$�ÓÙ(*I¦5E=UÎ!I?$:IU0�Ì�� I*Ó�=?�!=UÓ��Ù06Ö/G�=/V,$;CML=U�!Ï�G�$�Î�5EÒ*K;AlÞRÒ!$DÏ 0l�6I!0 Öc57CW�6=?�!=U�*Ï�=?K�570]I?0�Ì��657$�01�!$�06Öc5EÒ!$;K7$�Ó�=?�!=UÓ��*Al2R�.5WÌWÒ�IU$��6CE�!=?�!ÏÑa=UI?I%G�=JK7Ì�0.< $�C�57Ò!$@Ó�=?�!=UÓ�(!Ó�0 Ö/ÑaÒ*�.5E$�<1$�C4P*� K7=U�¾57Ò*$DÑ4$�=?Ï Òl5EKR� C7$D=U�!=U57=J�6I?I?Ð�Î!IJ� Ìr$�GcA>áh�Ke5E0>ÌWÒ9� KM57=JÌ�IU$��6CE�!=U�*Ï*-c5EÒ!$��!01=?K7$�Î!CE$;K7$��l5]=U�ú5EÒ!$�(!Î�G!�657$;K�Ì�� �åCE$;K7(!IU5Ù=?�ú57Ò!$�Ñ4$�=?Ï Òl5WK�e(!Ó�Î!=U�*Ïå=?�l570 57Ò!$�P*�1KM=?�ç0 Ö:� �!065EÒ!$�C�-vÎ,01KEK7=UP!I?ÐçG�$;$�Î,$�C�-®IU0�Ì�� IRÓ�=?�!=?ÓÙ(!Ó�A�ÞRÒ!=JK�Ò*�1KP9$;$��#G�$�Ó�01�*Ke5ECE�657$�Gº=?�#Ìr$�C75E� =U��K7=UÓ�Î!I?=¦â9$;G�Ì��1KM$�K � ��"!- � � �[A?l570�ÌWÒ*�1Ke5E=?ÌDIU$��6CE�!=?�!Ï�=JKa�6IJK70�(*K7$rÖÝ(*I%ÑaÒ!$��¾57Ò*$DÖÝ(*�*Ìn5E=U01�3P,$�=?�!Ï�Ó�0�G�$�I?$;G¾=?KaÌWÒ9�6�!Ï L=U�!Ïã0.< $;C�57=?Ó�$ -R�çÜl(!=U57$ Ì�0 Ó�Ó�0 ��KEÌr$;�*�6CE=U0ç=U��=?�*G�(9Ke5EC7=J�6I:�6Î!Î!I?=JÌ��.5E=U01�*K�ÑaÒ*$�CE$Á57Ò*$G!�.5W��G�=JKe5EC7=?P!(�5E=U01�ÁÌWÒ*�6�!Ï1$;KRÏ1CE�1G�(*�6I?I?Ð�0.< $;Ca57=?Ó�$ �Ý$1A Ï9A9G�(!$]570�Ñ4$;� C�6�*G¾5E$;�6CD06Ö®57Ò*$Ó��1ÌWÒ!=?�!$;K �nAcáOÖ)57Ò!$�I?$;� C7�!=?�!ϾÓ��1ÌWÒ!=?�!$�G�0>$;K@�!0 5�G!$r57$�Ìn5]�6�9GÁÖÝ01IUI?0.Ñß57Ò*$�ÌWÒ*�6�!Ï1$�=U5�=?K=UÓ�Î,01KEKM=?P!I?$5E0�I?$;� C7�º57Ò!$�G!�65E��Î!CE0 Î,$�CEIUÐ��6�9G�IJ�6CEÏ $:Ï $��*$�CW�6I?=Uë��.57=?0 ��$;C7CE0 CWK�Ña=?I?IcC7$�KM(!IU5;Aäx=¦5EÒzP*�.5WÌWÒúIU$��6CE�!=U�*Ï*-%ÌWÒ*� �!Ï $�K�Ï 0�(!�*G!$r57$�Ìn5E$;Gz� �*GåÑ)$�01P�5E� =U�¼CW�.5EÒ!$�C�P*�1GúC7$�KM(*I¦5WKKM=?�*Ìr$¾Ñ)$��6CE$�I?=UÍ1$�I?Ðå570ú�'< $�CW�6Ï1$�0.< $;CÙKM$;< $;CE� I�CE(!IU$�K�-�ÑaÒ!$;C7$�� K�0 ��LOIU=?�!$¾IU$��6CE�!=U�*Ï��å=UÖ0 Î,$�CW�.57$�G#Î!CE0 Î,$�CEIUÐ �ÀKM$;$ÙP,$�I?0.ÑÚ=U�åKM$�Ìn5E=U01� � A!(Q����Ña=?IUIv57CW� ÌWÍ35EÒ!$�ÌWÒ9�6�!Ï1$;K� �*GÁÐ>=?$�IJGÏ 0>0�G¾� Î!Î!CE0'æ>=?Ó��.5E=U01�ºCE$;K7(!I¦5WK�A

Û$�KMÎ!=U57$57Ò!$:� G�<.� �15W�6Ï1$;K®06Ö/KM570�ÌWÒ*�1Ke5E=?ÌRI?$;� C7�*=U�!Ï9-65EÒ!$�CE$�6CE$KM57=?IUI,CE$;� K70 �9K®ÑaÒlÐ�0 �*$Ó�=?Ï Òl5Ì�0 �*K7=?G!$�CR(*K7=U�!Ï�P*�65EÌWÒ�I?$;�6CE�!=?�!Ï C

8 9 #!�!3;���:4&%1 =�<>�+��!���-�/@B%1�2')3�"03�4�1A�&)01�*G�=U57=?0 �*KR0 Ö®Ì�0 �>< $;C7Ï1$��*Ì�$D� C7$:Ñ)$;IUI/(!�*G�$;CEKM570>0�GcAE ARQ �6�>Ð:�1Ì�Ì�$�I?$�CW�.57=?0 �@5E$;ÌWÒ!�*=?Ül(!$�K �Ý$1A Ï9A�Ì�0 �P�e(*Ï1�.5E$�Ï1CE�1G�=U$;�l5 ��01�!IUÐ:0 Î!L$;CE�657$@=?�3P9�.5EÌWÒ3I?$;�6CE�!=?�!Ï*A

�*ARÞRÒ*$�0 CE$r5E=?Ì;�6I�� �*�6I?Ð�KM=JK�0 Öa57Ò!$�Ñ4$�=?Ï Òl5�G!Ðl�9�6Ó�=?Ì;KÙ� �*G¼Ìr01�l<1$�CEÏ $;�*Ìr$CW�.5E$;Ka� C7$@KM=?Ó�Î!I?$�C�A

ÞRÒ!$;K7$º� G!<'� �l5E�6Ï1$;K�KM57$;Ó÷ÖÝCE0 Ó÷57Ò*$ºK7� Ó�$��!0 =JK7$�57Ò*�65ÙÓ��6Í1$ºKe5E0�ÌWÒ*� KM57=JÌ�IU$��6CE�!=?�!Ï� G�<.�6�l5W�6Ï $;0 (*K;A.ÞRÒ!=JK®�!0 =JKM$1-.ÑaÒ!=JÌWÒ�=JK�K70@Ì�C7=U57=JÌ�� I>ÖÝ0 C®â*�9G�=U�*Ï�P9$�5M57$;C�I?0>Ì;�6I�Ó�=?�!=UÓ���� I?K70Î!C7$;< $;�15WK@ÖÝ(!I?I4Ì�0 �>< $;C7Ï1$��*Ì�$Ù5E0#57Ò*$ºÓ�=U�*=UÓ�(!Ó�Aváh�*KM57$�� Gú0 ÖaÌ�0 �>< $;C7Ï1=U�*Ϻ5E0#57Ò!$¾$ræ!� Ìr5Ó�=?�!=?ÓÙ(!Ó�-�5EÒ!$�Ìr01�l<1$�CEÏ $;�*Ìr$KM5E� IUIJK40 (�5�G�(*$D5E0�57Ò!$@Ñ4$�=?Ï Òl5 �9(*Ìn5E(*�.5E=U01�*K�A!ÞRÒ*$@K7=?ë�$:06Ö57Ò!$ �*(*Ìn5E(*�.5E=U01�*KÙG!$�Î,$��*G¼01�ú57Ò!$¾G�$�Ï1C7$;$�06Ö��!01=?K7$�06ÖR57Ò*$¾KM570�ÌWÒ*� KM57=JÌ�(!Î�G!�.5E$;K;A�ÞRÒ*$<'� C7=J�6�9Ìr$º0 Ö5EÒ!$ �9(*Ìn5E(*�.5E=U01�*K��6CE0 (!�9Gú57Ò!$�I?0�Ì��6I4Ó�=U�*=UÓ�(!Ó =JK�Î!C701Î901CM5E=U01�*�6I�570å57Ò*$IU$��6CE�!=U�*Ï3CW�.5E$ 0 � E **- E (�- & �OA0?>0#=?�å01CEG!$�C@570#CE$;G!(*Ìr$�57Ò*$ �*(*Ìr57(*�657=?0 �*K�Ñ4$�Ì�� �å$;=¦5EÒ!$�CG�$;Ì�C7$�� K7$G�À�6�!�*$;�6IU�v57Ò*$�IU$��6CE�!=?�!Ï�CE�657$R0 C�Ò*�'< $a�6���1G!�6Î�5E=U<1$aP*�.5WÌWÒ�KM=?ë�$1A6áh��57Ò!$;0 CEÐ � � �*-� � - � &!- � " ��=U54=JKRKMÒ!0.Ña��57Ò*�65)5EÒ!$D01Î�57=?Ó��6I��6�!�*$;�6I?=?�!Ï�KEÌWÒ!$;G!(!IU$:06Öc5EÒ!$:IU$��6CE�!=U�*ÏÙCW�.57$D=?K06Ö�57Ò!$@ÖÝ01C7Ó

0����. �%� E �

ÑaÒ!$�CE$ .®=?K�57Ò!$D�>(!ÓÙP,$�C)06ÖcÎ*�.5757$;C7�*K�Î!CE$;K7$��l57$�G��6�*G � =JK��]Ì�0 �*KM5E� �l5;A áh��Î*CE�1Ìn57=JÌr$1-65EÒ!=?KÓ��'кP9$@5E0l0�Ö»�1Ke5

Page 7: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

���!0 57Ò!$;C�Ó�$r5EÒ!0�G�570�C7$;Ó�0.< $a�!01=?K7$�=JK�5E0Ù(*K7$ºè7Ó�=?�!=UL[P9�.5EÌWÒ*$;KEê!-.5EÒ*�.54=?K;-lKM5E� CM5)Ña=¦5EÒ�#KMÓ��6I?I�P9�.5EÌWÒ¼K7=Uë;$º�6�9G =?�*Ì�C7$�� K7$�57Ò!$ºK7=?ë�$º�1K@57CW�6=?�!=?�!Ï#Î!CE0>Ì�$�$�G!K�AvQ��6I?IU$;CÙG�=JK7Ì�(*KEKM$�K0 �!$�Ó�$r5EÒ!0�G ÖÝ01C]G�0 =?�!Ͼ57Ò!=JK � E " ���6�*G¼BDC7C � �"� ��G!=?KEÌr(*KEK7$;K:57Ò*=?K:ÖÝ01C�IU=?�!$;� C�Î!CE0 P!I?$�Ó�K;AØ�0.Ñ4$�< $;C;-lG�$;Ì�=?G!=U�!ÏÙ57Ò!$:CW�.57$:�.5RÑaÒ!=JÌWÒ�570�=U�9ÌrCE$;� K7$a5EÒ!$DP9�.5EÌWÒ¾KM=?ë�$:�6�*GºÑaÒ!=?ÌWÒº=U�!Î*(�5EK570�=U�*Ì�IU(9G�$@=U�¾57Ò!$�K7Ó��6I?I,P9�.5EÌWÒ*$;K4=?Ka�1KRG�=¦Õ�Ì�(!I¦5��1K4G!$r57$;C7Ó�=?�!=?�!Ï�57Ò!$:Î!CE0 Î,$�CRIU$��6CE�!=?�!ÏCE�657$ A A V�$;Ìr57=?< $�I?Ð357Ò!$�KM=?ë�$�06Ö�5EÒ!$ÙI?$;� C7�!=?�!Ï�CW�.5E$Ù=?� KM570�ÌWÒ*� KM57=JÌ�I?$;� C7�!=?�!ϺÌ�0 CEC7$�KMÎ,0 �9G!K570�57Ò*$:CE$;K7Î,$;Ìn5E=U<1$�KM=?ë�$�0 Ö/57Ò*$�Ó�=?�!=%P9�.5EÌWÒ/A

��065E$º�6IJK70�57Ò*�65]57Ò!$¾Î!CE0 P!I?$�Ó÷06ÖaC7$;Ó�0.<l=?�!Ï�5EÒ!$��!01=?K7$�=?�ú57Ò!$3G!�.5W�#Ó��'Ð P,$�IU$�K7KÌrCE=¦5E=?Ì;�6Ic5EÒ*�6��0 �*$@57Ò!=?�!Í�KaP,$;Ì�� (*K7$�06Ö®Ï $��*$�CW�6I?=Uë��.57=?0 �/A9BD< $�C757CW�6=?�!=?�!Ï�Ó��'к0>Ì;Ìr(!CIU01�!ÏP9$�ÖÝ0 CE$D5EÒ!$��!01=?K7$@C7$;Ï =?Ó�$@=JKa$�<1$��3C7$�� ÌWÒ!$�GcA���!0 57Ò!$;CD� G!<'� �l5E�6Ï1$�06Ö�P*�65EÌWÒ�5ECE� =U�!=?�!Ï�=JKa5EÒ*�.5:0 �!$�=?KD�6P!I?$�5E0º(*K7$ÙK7$;Ìr01�*G�0 CWG�$;CÓ�$�57Ò!0�G!K�5E0åK7Î,$�$;Gç57Ò*$3I?$;� C7�*=U�!ÏåÎ!CE0�Ìr$;KEK;A�?>$�Ìr01�*Gz01CEG�$;C�Ó�$�57Ò!0�G!K�K7Î9$;$;GãIU$��6CE�!=?�!ÏPlÐ $;KM57=?Ó��657=?�!Ï��!065��e(9Ke5�5EÒ!$�Ï CW� G�=?$��l5@P!(�5Ù� I?K70º5EÒ!$�Ìr(*C7<.�.5E(!CE$�0 Ö�5EÒ!$�Ìr0lKe5]K7(!CMÖ»�1Ìr$1A8:=U<1$��ú57Ò*$ºÌr(!CE<.�.5E(!C7$1-%0 �*$ºÌ�� �å$�Ke5E=UÓ��657$�5EÒ!$º�6Î*Î!C70'æ�=?Ó��.57$�I?0�Ì��.5E=U01�ú06ÖR57Ò!$3� Ìr57(*� IÓ�=?�!=?ÓÙ(!Ó�A

Û$�KMÎ!=U57$�57Ò!$�1G�<.�6�l5E� Ï $�K�06Ö�P*�.5WÌWÒ�(!Î�G!�.5E$;K;-1KM570�ÌWÒ*� KM57=JÌRI?$;�6CE�!=?�!Ï�=JK�KM57=?IUI906Ö×57$;��57Ò*$Î!C7$�ÖÝ$�CEC7$�G�Ó�$r5EÒ!0�GºÎ*� CM5E=?Ì�(!IJ�6CEIUÐ�ÑaÒ!$;�¾G�$��6I?=U�!Ï�Ña=U57Òº<1$�CEÐ�I?� C7Ï1$DG!�65E��KM$�5EK4P9$�Ì�� (*KM$:=U5=?K�K7=?Ó�Î*IUÐ�Ó�(*ÌWÒ3Ö»� KM57$;C;A

����� ���*�� "�3C4 ���C%������ � .0%1 ��$r5eÑ40 CEÍ�K]I?$;� C7�ú5EÒ!$ºÖ»�1Ke5E$;KM5ÙÖÝCE0 Ó 57Ò*$¾Ó�01KM5�(*�!$ræ�Î,$;Ìn5E$;GãK7� Ó�Î*IU$1A�ÞRÒ!$;C7$�ÖÝ0 CE$ -v=¦5�=?K� G�<>=JK7� P!IU$�570 ÌWÒ*0l0lKM$3�ÁKE�6Ó�Î!I?$¾�.5�$;�1ÌWÒ¼=¦5E$�CW�.5E=U01�¼57Ò*�65�=JK]5EÒ!$3Ó�0lKe5�(!��Ö»� Ó�=?I?=?� CÙ5E057Ò!$úKMÐ�KM57$;Ó3A��0657$1-457Ò!=JK#�6Î!Î*IU=?$;K¾0 �!I?Ðì570ìKM570�ÌWÒ*� KM57=JÌÁI?$;� C7�*=U�!ÏTKM=?�*Ì�$ 5EÒ!$ 01CEG!$�C306Ö=U�!Î*(�5�Î!CE$;K7$��l5E�657=?0 �ç=JK�=?C7CE$�I?$�<.�6�l5�ÖÝ0 C�P*�65EÌWÒ,+.A)BÖ:Ìr01(!CWKM$1-v5EÒ!$�CE$�=?K��!0¼K7=UÓ�Î!I?$�Ñ4�'Ð570åÍl�*0.Ñ ÑaÒ!=?ÌWÒç=U�!Î*(�5EK�� C7$¾=U��ÖÝ01C7Ó��657=?0 �zCE=?ÌWÒ/-vÒ!0.Ñ4$�<1$�C�-v� < $�CEмKM=?Ó�Î!I?$º5EC7=JÌWÍå57Ò*�65ÌrCE(*G�$�I?Ð =?Ó�Î!I?$�Ó�$��l5EK]57Ò!=JK�=JG�$;��=JK�5E0ÁKM=?Ó�Î!I?Ð ÌWÒ!0>01K7$�K7(*Ì�Ì�$;KEKM=?< $�$ræ!� Ó�Î*IU$�K:57Ò*�65��6CE$ÖÝC701Ó þ.ÿ � �rü �����4ÌrIJ� KEKM$�K�KM=?�*Ì�$�57CW�6=?�!=?�!Ï#$ræ!�6Ó�Î!I?$;K]P9$;IU01�!Ï =?�!Ï�570�5EÒ!$ºKE�6Ó�$�ÌrIJ� KEK�Ña=?IUIÓ�0lKe5aI?=UÍ1$�I?оÌ�0 �l5E� =U�3KM=?Ó�=UIJ�6Ca=?��ÖÝ0 CEÓ��.5E=U01�%A���!0 57Ò!$;C�Ò!$;(!C7=JKM57=JÌaÖÝ0 C �e(*G�Ï1=U�*ÏÙÒ!0.ÑìÓÙ(9ÌWÒ��!$�Ñx=?��ÖÝ01C7Ó��.5E=U01����5ECE� =U�!=?�!Ï]$�æ!�6Ó�Î!I?$Ìr0 �l5W�6=?�*K/=JKv570D$ræ!�6Ó�=?�!$�5EÒ!$4$�CEC701C/P,$r5eÑ4$�$��]57Ò!$R�!$�5eÑ)01C7Í:0 (�5EÎ!(�5��6�*G]57Ò!$)5E� C7Ï1$r5�<'� IU(*$ÑaÒ!$���57Ò*=?K�=?�!Î!(!5�=?K®Î*C7$�KM$;�15E$;GcA �ìI?� C7Ï1$)$;C7CE0 C®=?�*G�=JÌ��.5E$;K®5EÒ*�.5�5EÒ!=JK�=U�*Î!(�5�Ò*�1K��!065�P,$�$;�IU$��6CE�!$;G#P>Ð357Ò*$��*$r5eÑ40 CEÍ3� �*GÁK703Ì�0 �l5E� =U�*KD�ºI?065:06Ö��*$�ÑÚ=?��ÖÝ0 CEÓ��.5E=U01�%A,ÞRÒ*$�CE$rÖÝ0 CE$ -,=U5Ó�� Í $�KvK7$��*K7$)570:Î!C7$�KM$;�l5v5EÒ!=?K�=?�!Î!(�5�Ó�0 CE$�ÖÝCE$;Ül(!$;�l57I?Ð A BÖ,Ì�0 (!CWKM$1-;P>Ð#è7I?� C7Ï1$;êRÑ4$)Ó�$;� �C7$;I?�657=?< $�5E0�6I?I10 Ö�57Ò!$)065EÒ!$�C/57CW�6=?�!=U�*Ï�$ræ!�6Ó�Î!I?$;K;A��Kc5EÒ!$4�!$r5eÑ40 CEÍ�5ECE� =U�*K;-r5EÒ!$;K7$�CE$�IJ�.5E=U<1$$�CEC701CEKÑa=UI?I�ÌWÒ*�6�*Ï $��6�*G KM0�KMÒ*0 (!IJGÁ5EÒ!$�ÖÝC7$�Ül(!$��*Ì�Ð#06Ö�Î*C7$�KM$;�15W�.5E=U01�ÁÖÝ0 C��¾Î*�6C757=JÌr(*I?� C=U�!Î*(�5�Î*�.5757$�CE�%A � Ó�$�57Ò!0�Gç57Ò*�65�Ó�0�G�=Uâ*$;K�5EÒ!$¾Î*C701P*�6P!=?I?=¦5eÐú06Ö:�6Î!Î,$;� CE� �*Ìr$¾06Ö$�� ÌWÒÎ*�.5757$�CE�3=?K�Ì;�6I?IU$�G¾� � ��� � �!ý�nÿ/Trÿ��>û D ��� � �rA

� �C���& �%������� ��.�%1 ��," ����� ����"� *��� 3C>��&'� �2��"��&3 � �<3;��%13;��1A�?�Òl(���$Á57Ò!$ 57CW�6=?�!=U�*ÏzKM$�5¾KM0z57Ò*�65¾KM(9Ì�Ìr$�K7K7=?< $�57CW�6=?�!=?�!Ïz$�æ�� Ó�Î!IU$�K�*$�< $;C �ÝCW�6CE$�I?Ð"��P,$�I?0 �!Ï�5E0�57Ò!$]KE�6Ó�$�ÌrIJ� KEK�AE A���C7$�KM$;�l5�=?�!Î!(�5�$ræ!� Ó�Î*IU$�KÙ5EÒ*�.5�Î!CE0�G�(*Ìr$�� I?� C7Ï1$º$�CECE0 CÙÓ�01C7$ºÖÝC7$�LÜl(!$;�l57I?Ð�57Ò*� ��$ræ!�6Ó�Î!I?$;K45EÒ*�.5�Î!CE0�G�(*Ì�$@��K7Ó��6I?I%$�CEC701C;A

X�u®ol_�d�bM�1_Ebvikj:��olikfMo�^;bM\�� ik_Wj.seg�\rbe_�gh�1Z�Z�_n�Dikj@{l\�sefMo�Z]\W�D{�_�\r¿!_WfEse_n�:{.�bed��1j>�1d�¿Ù_Ebhbed;bi�°�seol_7be_ai¦g�\:ghik^;jli�ª>fn\�j.s�bM\�jl^;_ad;°%^;bM\;�1ik_Wj.s���\��k�l_WgW©

Page 8: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Ø�0.Ñ4$�< $;C;-®01�!$ÁÓÙ(9Ke5ºP,$ Ì;�6CE$rÖÝ(!I�ÑaÒ*$��xÎ,$�C757(!CEP!=?�!ϼ57Ò*$Á�!0 CEÓ��6IaÖÝCE$;Ül(!$;�*Ìr=?$;K�06Ö=U�!Î*(�5�$ræ!�6Ó�Î!I?$;K:P,$;Ì�� (*K7$�5EÒ!=JK�ÌWÒ*�6�!Ï1$;K57Ò!$�CE$�IJ�.5E=U<1$Ù=?Ó�Î,0 C75E� �*Ìr$�57Ò9�.5@57Ò!$��!$�5eÑ)01C7ÍÎ!I?�1Ìr$�K/01�ÙG�=/V,$;C7$;�15�$ræ!�6Ó�Î!I?$;K;A'ÞRÒ!=?KvÓ��'Ð@0 C/Ó��'Ð@�!0 5vP,$4G�$�KM=?CW�6P!I?$ A�î!0 C�$ræ!� Ó�Î*IU$1- � �>ÿ � � D � �*ÿ�� ����ý � � �¦ÿ��Eþ � �#þ ý��hý D ��� �Oý6ÿ��*ÿ��>û ��� ���kÿ���ü ,DWý�� � ��þ6ÿ �ý���Àü ��� P,$;Ì;�6(*K7$�01(�57I?=?$�CWKÌ��6�åÎ!CE0�G�(*Ìr$�IJ�6CEÏ $�$�CECE0 CWKÐ1$r5ÙK7Ò!0 (!IJG �!065]P,$�Î!CE$;K7$��l57$�GÁÖÝCE$;Ül(!$;�15EIUÐ1AvBD�å57Ò!$�0 57Ò!$;CÒ*�6�*G%-15EÒ!=JK�5E$;ÌWÒ!�!=JÜl(!$@Ì��6�ºP9$:Î*�6C757=JÌr(*I?� C7I?Ð�P,$��*$râ9Ì�=?� I*ÖÝ01C)P,0>01KM57=?�!Ï]5EÒ!$DÎ,$�C7ÖÝ0 CEÓ��6�*Ì�$ÖÝ0 Ca=?��ÖÝC7$�Ül(!$��l57I?к0�Ì;Ìr(!CEC7=?�!Ï�=U�*Î!(�5EK;-!$1A Ï9A 2'ë�2Ù=?��Î!Ò!0 �*$�Ó�$@C7$�Ìr0 Ï1�!=U57=?0 �

����� �A�&'� �!.0"�� "�3C4 ����%�� 3 ��*C��

&)0 �>< $;C7Ï1$��*Ì�$)=JK�(*K7(*� IUI?Ð]Ö»�1Ke5E$�C�=UÖ�57Ò!$�'<1$�CW�6Ï1$)0 Ö,$�� ÌWÒ�=U�*Î!(�5�<.�6CE=J�6P!I?$a0.< $;C®57Ò!$a5ECE� =U�!L=U�!Ï�K7$r5®=?K�Ì�IU0lKM$)570@ë;$�CE0*A'Þ/0�K7$�$)57Ò!=JK�-6Ì�0 �*K7=JG�$�C�57Ò*$4$�æ>57CE$�Ó�$aÌ�� K7$4ÑaÒ!$�CE$4� IUIl5EÒ!$R=U�!Î*(�5EK�6CE$)Î,01K7=¦5E=U<1$ A'äå$�=?Ï Òl5EKv570��DÎ*�6C757=JÌr(*I?� Cv�!0�G�$R=?�Ù5EÒ!$4â*CWKe5®Ñ)$;=UÏ1Òl5�IJ�'Ð $;C�� C7$)(!Î,G*�.57$�GÙP>Ð�6�º� Ó�01(!�l5�Î!CE0 Î,0 C757=?0 �*� I�570�� � ÑaÒ*$�CE$�@=?K�57Ò*$ �»KEÌ�� I?� C ��$�CEC701C��.5)57Ò*�65)�!0�G�$:�6�*G � =?K57Ò!$:=U�*Î!(�5R< $;Ìr5701C �»K7$�$:$;Ül(*�657=?0 �*K �#" ��� �*G�� � � �;�nA>äxÒ!$��3�6I?I90 Öc57Ò!$@Ìr01Ó�Î901�!$��l5EK)06Ö/� �=U�!Î*(�54<1$;Ìr570 C4�6CE$Î,01K7=¦5E=U<1$ ->�6I?I90 Öc57Ò!$:(!Î�G!�.5E$;K406Ö%Ñ4$�=?Ï Òl5WK�5EÒ*�.5)ÖÝ$�$;Gº=U�l5E0��]�*0>G!$DÑa=?IUIP9$�57Ò!$�KE�6Ó�$�K7=?Ï ���»=ÀA $ AcK7=UÏ1�0���Q�;�nA �K:�¾C7$�KM(!IU5;-957Ò*$;K7$�Ñ4$�=?Ï Òl5EK:Ì�� � 01�!IUÐ#� IUI®G�$;Ì�C7$�� K7$0 C� IUI%=?�*Ì�C7$�� K7$ � �Eû ��� � ��ü)ÖÝ01C��Ï =?< $;��=U�!Î*(�5DÎ*�65M5E$�CE�%A*ÞRÒ>(*K;-*=UÖ���Ñ4$�=?Ï Òl5�<1$;Ìn5E0 C�Ó�(*KM5ÌWÒ*�6�!Ï1$ºG�=?C7$�Ìn5E=U01�z=¦5�Ì�� �z0 �!I?ÐúG�0 K70ÁP>Ðúë�=?Ï ë;� Ï Ï1=U�!Ï#ÑaÒ*=?ÌWÒz=JK�=U�!$�Õ�Ìr=?$��l5�� �*Gú57Ò>(*K< $�CEÐ�K7IU0.Ñ�A

áh�º5EÒ!$]�6P,0.< $@$ræ!� Ó�Î*IU$1->57Ò!$�=?�!Î!(�5WKaÑ4$�CE$:� IUI%Î,01K7=U57=?< $ A!Ø0.Ñ)$;< $;C;->=?�3Ï1$��!$;CE� IÀ-�� �lÐKMÒ!=UÖ×5�06Ö)57Ò!$��'< $;CE� Ï $]=?�!Î!(�5]�'ÑR�'Ð3ÖÝC701Ó�ë�$�CE0ºÑa=?I?I�P!=?�1KD57Ò!$�(!Î�G!�.5E$;K@=?� �¾Î*�6C757=JÌr(*I?� CG�=UCE$;Ìr57=?0 �Á� �*G357Ò>(*KDKMI?0.Ñ G!0.Ña�#IU$��6CE�!=U�*Ï*A*ÞRÒ!$;C7$�ÖÝ0 CE$ -!=U5D=JK�Ï10>0>G35E0ºK7Ò!=¦Ö×5D57Ò!$Ù=U�!Î*(�5EKKM0�5EÒ*�.5D57Ò!$Ù�'<1$�CW�6Ï1$:0.<1$�CR5EÒ!$�57CW�6=?�!=?�!ϺK7$r5=JKÌ�IU0lKM$@5E0�ë�$;C709AÁÞRÒ!=JK�Ò!$;(!C7=JKM57=JÌ]KMÒ*0 (!IJGP9$º� Î!Î!I?=U$�G¼�.5�� IUI)I?�'Ð1$�CWK@ÑaÒ!=JÌWÒ¼Ó�$��6�*K]57Ò*�65ÙÑ4$�Ñ4� �l5�57Ò!$º�'<1$�CW�6Ï1$�0 Ö45EÒ!$ ��� �!��� ��06Ö4�¾�!0�G�$�570�P,$�Ì�IU0lKM$Ù570�ë�$;C70¾P9$�Ì�� (*KM$�57Ò*$;K7$�01(�57Î*(�5EK�� C7$Ù57Ò!$�=?�!Î!(�5WK:570357Ò!$��!$�æ>5I?�'Ð1$�C � ��- �OA/ÞRÒ!=JK�Î!CE0 P!I?$�Ó÷Ì��6�úP,$�� G!G!C7$�K7K7$;G P>Ð Ìr0>01CEG�=?�*�657=?�!Ï�Ò!0.ÑÚ5EÒ!$�=U�*Î!(�5EK��6CE$57CW�6�*KMÖÝ0 CEÓ�$;GúÑa=U57Ò¼5EÒ!$¾ÌWÒ!01=?Ì�$�06ÖK7=?Ï Ó�0 =JG!�6I4� Ìr57=?<'�657=?0 �úÖÝ(!�9Ìn57=?0 �/A®Ø$�CE$�Ñ4$ºG�=JK7Ì�(*K7K57Ò!$�=?�!Î!(�5a5ECE� �*KMÖÝ0 CEÓ��657=?0 �%A�ÞRÒ!$]G!=?KEÌr(*KEK7=U01�¾0 Ö/57Ò*$]KM=?Ï Ó�0 =JGºÖÝ0 I?IU0.Ñ�K;A

&)0 �>< $;C7Ï1$��*Ì�$�=JK�Ö»�1Ke5E$�C]�!0 5Ù0 �*IUÐ =¦ÖR57Ò*$�=U�*Î!(�5EK��6CE$�KMÒ*=¦Ö×5E$;Gz�1K]G�$;KEÌrCE=?P9$�Gå� P90.<1$P!(�5a�6IJKM0]=UÖ%5EÒ!$�Ð��6CE$DKEÌ�� IU$�G�KM0Ù57Ò*�654� IUI,Ò*�'< $D�6P,0 (�5457Ò!$:K7� Ó�$DÌr0.<.� C7=J�6�*Ì�$ - � , ->ÑaÒ!$�CE$

� ,�� �I9

J�6Kv+�� �, � F 5 �%� � �

Ø�$�CE$ - I =?K�57Ò!$D�>(!ÓÙP,$�C)06Öc57CW�6=?�!=?�!Ï�$ræ!� Ó�Î*IU$�K�-�� , =JK�57Ò*$DÌr0.<.� C7=J�6�*Ì�$a06Ö�57Ò*$� (�� =?�!Î!(�5<'� C7=J�6P*IU$� �*G �, =JK�57Ò!$�� (�� Ì�0 Ó�Î901�!$��l5406Öc57Ò*$ � (�� 57CW�6=?�!=?�!ÏÙ$�æ!�6Ó�Î!I?$ A$?�Ì�� IU=?�!ÏÙK7Î9$;$;G!KIU$��6CE�!=U�*Ï�P9$�Ì��6(9KM$D=¦54Ò!$�I?Î*K�5E0ÙP*� I?� �*Ìr$a01(�5�5EÒ!$CW�.5E$�.5)ÑaÒ!=?ÌWÒ�57Ò!$DÑ)$;=UÏ1Ò15WK�Ìr01�!�!$�Ìn57$�G570357Ò!$�=?�!Î!(�5]�*0>G!$;K:IU$��6CE�%AcÞRÒ!$�<.�6I?(!$�06Ö�5EÒ!$�Ìr0.<.�6CE=J�6�*Ì�$�K7Ò!0 (!IJG P9$�Ó��.5WÌWÒ!$;G Ña=¦5EÒ57Ò*�65�0 Öa57Ò!$�KM=?Ï Ó�01=?Gú(*K7$;G%A®î*0 CÙ57Ò!$�KM=?Ï Ó�01=?GúÏ1=U<1$��zP,$�I?0.Ñ�-�� Ìr0.<.�6CE=J�6�*Ì�$�06Ö �º=JK��Ï 0>0�G¾ÌWÒ!01=?Ì�$ A

ÞRÒ!$�$ræ!Ìr$;Î�57=?0 �35E0ºK7Ì;�6I?=U�*Ï�� IUI�Ìr0.<.�6CE=J�6�*Ì�$;K)570�5EÒ!$ÙKE�6Ó�$�<.�6I?(!$]0�Ì�Ì�(!CEKRÑaÒ*$��#=U5=?KÍl�*0.Ña� 5EÒ*�.5ÙK70 Ó�$�=U�*Î!(�5EKÙ�6CE$�0 Ö4I?$;KEK�K7=UÏ1�!=¦â,Ì��6�9Ìr$�57Ò*� � 0 57Ò!$;CEK;Acáh�¼KM(9ÌWÒå�#Ì;� K7$ -c=U5Ì��6��P,$]P,$��!$�â9Ìr=J�6Ic5E0ºKEÌ��6I?$:57Ò!$ÙIU$�K7K�K7=?Ï �!=Uâ9Ì�� �l5�=U�!Î*(�5EKDG�0.Ña�3K70�57Ò9�.5�5EÒ!$�Ð��6CE$�èMI?$;KEK<l=JK7=UP!I?$;ê�5E0�5EÒ!$�I?$;�6CE�!=?�!Ï�Î!CE0>Ì�$;KEK�A

Page 9: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

� ')�!3� �>��&'� "03�4 ����%�� 3 ��*C�� �1ARÞRÒ*$D�'< $;CE� Ï $406Ö%$�� ÌWÒ�=?�!Î!(�54<'� C7=J�6P*IU$�0.<1$�C�5EÒ!$�5ECE� =U�*=U�!Ï�KM$�5)KMÒ*0 (!IJG�P,$DÌrI?01K7$5E0�ë�$�CE0*AE A�?!Ì��6I?$@=U�*Î!(�5�<.�6CE=?� P!I?$;KaK70�5EÒ*�.5a5EÒ!$�=?C�Ìr0.<.� C7=J�6�*Ì�$;K4�6CE$��6P,0 (�5a5EÒ!$]K7� Ó�$ A

�*A4áh�*Î!(�5�<.�6CE=?� P!I?$;KaK7Ò!0 (!IJG¾P9$�(!�9Ìr0 CECE$�IJ�.57$�G�=UÖ�Î,01KEK7=UP!I?$ AÞRÒ!$��6P,0.< $�5eÑ)0357CE=JÌWÍ>K:0 ÖRK7Ò!=UÖ×57=?�!ÏÁ�6�*GåKEÌ��6I?=?�!Ϻ5EÒ!$�=?�!Î!(�5WK]�6CE$�Ü1(*=¦5E$�KM=?Ó�Î!IU$�5E0

=UÓ�Î!I?$�Ó�$��l5�A"���!0 57Ò!$;CR57CE=?ÌWÍ�5EÒ*�.5a=JKaÜ1(*=¦5E$:$6V,$�Ìn5E=U<1$:P*(�5aÓ�01C7$@G�=UÕ�Ìr(!IU5a570�=UÓ�Î!I?$�Ó�$;�15=?K570ºG!$;Ìr01C7CE$�IJ�.5E$@57Ò!$�=U�*Î!(�5EK;A%&)0 �*K7=?G!$�C�5EÒ!$�K7=?Ó�Î*IU$��!$r5eÑ40 CEͺ=?� î�=UÏ1(!CE$ E A9áOÖ�=U�!Î*(�5EK�6CE$Ù(!�*Ì�0 CEC7$;I?�657$�G357Ò!$;� =U5:=JK:Î90lK7K7=?P!IU$�570�K70 I?< $�ÖÝ01CD5EÒ!$�<'� IU(*$�0 Ö � + 5EÒ*�.5�Ó�=U�*=UÓ�=?ë�$;K57Ò!$�$;C7CE0 C@Ña=U57Ò!01(�5Ù�6�>Ð Ì�0 �*Ì�$�CE�ÁÖÝ0 C � F -/�6�9G <>=JÌr$�< $;CEKE�!A�áh�å065EÒ!$�C�Ñ40 CWG!K;-�57Ò!$�5eÑ40<'� C7=J�6P*IU$�K��6CE$a=U�9G�$�Î,$��*G!$��l5��Ý57Ò!$:KMÐ�KM57$�Ó 06Öc$�Ü1(9�.57=?0 �9K�=JK�G�=J�6Ï10 �*� I8�rA6äx=¦5EÒºÌr01C7CE$�IJ�.5E$;G=U�!Î*(�5EK;-101�!$�Ó�(*Ke5RKM01IU<1$RÖÝ0 C)P90 57ÒºK7=UÓ�(!IU5E�6�*$�0 (9KMI?Ð�ÑaÒ*=?ÌWÒ�=JK)�]ÓÙ(*ÌWÒ�Ò*� CEG!$�C�Î!CE0 P*IU$;Ó3A��CE=U�*Ì�=UÎ9�6IRÌr0 Ó�Î,0 �!$;�l5�� �*�6I?Ð�KM=JK �À�6IJKM0 Í>�!0.Ña�ç� K�5EÒ!$3H�� C7Ò>(!�*$���Lh"%0>$�<1$�$ræ�Î*� �*K7=U01� �Ì��6�3P,$�(*KM$�G¾570�C7$;Ó�0.< $ �kÿ�� �Wý.üaÌ�0 CEC7$;I?�657=?0 �*K)=U��=?�!Î!(�5WK � � � �OA

áh�!Î!(�5WK�57Ò9�.53� C7$#I?=?�!$;� C7I?ÐTG!$�Î,$��*G�$;�l5 �×5EÒ!$ $�æ>57CE$�Ó�$ÁÌ��1KM$Á0 Ö]Ìr0 CECE$�IJ�.57=?0 � ��Ó��'Ð�6IJKM0@Î!CE0�G�(*Ìr$�G!$�Ï $;�!$�CW� Ì�=U$�K�ÑaÒ!=?ÌWÒ�Ó��'ÐÙK7I?0.ÑçI?$;�6CE�!=?�!Ï*A1&)01�*K7=?G�$;C�57Ò*$�Ì�� K7$aÑaÒ!$;C7$R0 �*$=U�!Î*(�5�=JK��6I?Ñ4�'Ð�Kv5eÑa=JÌr$R57Ò*$�065EÒ!$�C�=?�!Î!(�5��� F � E + �nA1ÞRÒ!$a�!$r5eÑ40 CEÍ]01(�57Î!(!5�=?K�Ìr0 �9Ke5W�6�l5�6I?0 �!Ï�IU=?�!$�K � F ��� F �%��� E �%� + -�ÑaÒ!$;C7$���=JKa��Ì�0 �*KM5E� �l5;A�ÞRÒ>(*K�->5EÒ!$@Ï CW� G�=?$��l5R=?KRë;$�CE0�6I?0 �!Ï@5EÒ!$;K7$DG�=?CE$;Ìn5E=U01�*K:�ÀKM$;$Dî�=?Ï (!CE$ E �rA1Q#0.<>=?�!ÏÙ� IU01�!Ï@57Ò!$�KM$DIU=?�!$�K�Ò*�1K��6P9KM01IU(�5E$�I?Ð��*0$4V�$;Ìn5�01�ÁIU$��6CE�!=?�!Ï*A�äå$�� C7$Ù57CEÐl=?�!Ͼ570�KM01IU<1$Ù=?� E LOÛ ÑaÒ*�65:=JK:$4V�$;Ìn5E=U<1$�I?Ð#0 �!I?ÐÁ����LOÛÎ!C701P!I?$�Ó�AúáeG�$;� IUI?Ð�Ñ)$�Ñ4� �l5570¾C7$;Ó�0.< $]01�!$�0 Ö�57Ò!$�=?�!Î!(�5WK@ÑaÒ!=?ÌWÒ Ña=UI?I�G�$;Ì�C7$�� K7$]57Ò*$KM=?ë�$@06Ö�57Ò*$��!$r5eÑ40 CEÍ�A

îv=?Ï (*C7$�¾KMÒ*0.Ñ�K�5EÒ!$�$;�l57=?C7$�Î!C70�Ì�$;KEK�06Ö�5ECE� �*KMÖÝ0 CEÓ�=?�!Ï�=?�!Î!(!5EK;A�ÞRÒ*$�KM57$;Î*KD� C7$B� ���KMÒ!=UÖ×5=?�!Î!(�5WK�K70�57Ò!$�Ó�$;� ��=?Kaë;$�CE0*- � E �RG�$�Ìr0 CECE$�IJ�.57$:=?�!Î!(�5WK�-,�6�*G ��� �R$;Ül(*� IU=?ë�$�Ìr0.<.� C7=UL�6�*Ì�$;K;A

z1 z2

y

ω1 ω2

ω2

ω1

Lines ofconstant E

L<M7N ¹��*¹ y9ikjl_W\�be����� _Wq�_Wjl�1_Wj.s�ikjlq1�1segW©

Page 10: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

MeanCancellation

KL-Expansion

CovarianceEqualization

L<M7N ¹��!¹ u*bM\;jlgO°?d;beZ]\rsei¦d;j�d;°%ikj1ql�1segW©

��� � � ��% ��"04 �<" 9

��0 �*IU=?�!$;� Cº� Ìr57=?<.�.57=?0 �ìÖÝ(*�*Ìn5E=U01�*Kº� C7$#ÑaÒ*�65ºÏ =?< $#�*$�(!CW�6ID�!$r5eÑ40 CEÍ�K�57Ò!$;=UC¾�!0 �!I?=?�!$;� CÌ��6Î9�6P!=?IU=U57=?$;K;A1BD�!$R06Ö,57Ò*$aÓ�0lKe5�Ì�0 Ó�Ó�0 ��ÖÝ0 CEÓ�K®0 Ö,�1Ìn57=?<.�.5E=U01�ÙÖÝ(!�9Ìn57=?0 ��=JK®57Ò*$�KM=?Ï Ó�0 =JGÑaÒ!=?ÌWÒÙ=?K��aÓ�0 �!0 5701�!=?Ì;�6I?IUÐ:=?�*ÌrCE$;�1KM=?�!ÏRÖÝ(!�*Ìr57=?0 �Ù57Ò*�65�� K7Ð>Ó�Î�570 57$;Kv�.5®KM01Ó�$�â*�!=U57$)<'� IU(*$� K���� =JK��6Î!Î*C70l� ÌWÒ!$�GcA�ÞRÒ!$#Ó�01KM5�Ì�0 Ó�Ó�0 �T$ræ!�6Ó�Î!I?$;K�� C7$35EÒ!$ÁKe5W�6�*G*�6CWGçI?0 Ï1=?KM57=JÌÖÝ(!�*Ìn5E=U01� ��� � ��� ���"� � 7�� �� �:�6�9G Ò>Ð>Î9$;C7P,0 I?=JÌ]5W�6�!Ï1$��l5��� � �G�Ú5E� �!Ò � � �@K7Ò!0.Ña� =?�îv=?Ï (*C7$ � A ?>=UÏ1Ó�0 =JG!K®57Ò9�.5)� C7$�K7Ð>Ó�Ó�$�57CE=?Ìa� P901(�5�57Ò*$�0 CE=UÏ1=U�@�»$ A Ï*A1K7$�$�î�=?Ï (!CE$ � P ���6CE$Î!C7$�ÖÝ$�CEC7$�G]ÖÝ01C®57Ò!$DK7� Ó�$RCE$;�1KM01�Ù5EÒ*�.5�=?�!Î!(�5WK�K7Ò!0 (!IJG�P9$�!0 CEÓ��6I?=Uë;$;Gc-.�*� Ó�$�I?Ð -.P,$;Ì�� (*K7$57Ò!$;Ðç�6CE$ºÓ�0 CE$ºI?=?Í $�I?Ðú570åÎ!CE0�G�(*Ìr$30 (!57Î!(�5WK@�ÝÑaÒ!=JÌWÒì�6CE$ ÿ������ ���5E0 5EÒ!$��!$ræ>5�I?�'Ð1$�C-�57Ò*�654�6CE$a0 �¾�'< $�CW�6Ï1$4Ì�IU0lKM$�570]ë;$�CE0*A1ÞRÒ*=?K�=?K)=U�¾Ìr0 �l5ECE�1Ke5�- K7�'Ð1-.570Ù57Ò!$DIU01Ï =JKe5E=?Ì4ÖÝ(*�*Ìn5E=U01�ÑaÒ!01K7$@0 (�5EÎ!(�5EK�6CE$:� IUÑR�'Ð�K)Î,01K7=U57=?< $��6�*G�K70�ÓÙ(*KM5aÒ*�'< $���Ó�$;�6�¾57Ò9�.5�=JKaÎ90lKM=U57=?< $1A

��"04 �&" 9 �1A�?�ÐlÓ�Ó�$r5EC7=J̺K7=?Ï Ó�0 =JG!K]K7(*ÌWÒz� KÙÒ>ÐlÎ,$�CEP,0 I?=?Ì�5W�6�!Ï1$��l5Ù0 Ö×57$��ãÌr0 �><1$�CEÏ $�Ö»� KM57$;C5EÒ*�6�35EÒ!$]Ke5W�6�*G*�6CWG�I?0 Ï =JKM57=JÌÖÝ(*�*Ìn5E=U01�%AE A� CE$;Ìr01Ó�Ó�$;�*G�$�GzKM=?Ï Ó�01=?G � � - �)=JK6C ��� � � � �X5!(�� "�- 5W�6�!Ò� F V �� A<?>=?�*Ì�$�57Ò*$5W�6�!ÒÁÖÝ(!�*Ìn5E=U01�Á=JKDK70 Ó�$r5E=UÓ�$�KÌr01Ó�Î!(�5E�657=?0 �*� IUI?Ð3$ræ�Î9$;�*K7=U<1$ -��6�Á�6Î*Î!C70'æ�=?Ó��.L5E=U01��06Öv=U5�P>к��CW�.57=?0�06Ö®Î901IUÐ>�!01Ó�=J�6IJKRÌ�� �3P,$�(*K7$;G¾=U�9Ke5E$;� G%A

�*A�?�0 Ó�$r5E=UÓ�$;K�=U5�=JK�Ò!$�I?Î�ÖÝ(!I>5E0:�1G!GÙ�:K7Ó�� IUIlI?=U�!$��6C�57$;C7Ó�-'$ A Ï*A���� � � �x5E�6�*Ò0� � ��7 � K70�� K4570��'<10 =JG �9�65KMÎ,065WK�A

Page 11: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

-6 -4 -2 2 4 6

0.2

0.4

0.6

0.8

1

-3 -2 -1 1 2 3

-1.5

-1

-0.5

0.5

1

1.5

(a) (b)

L<M7N ¹��,¹�� \����4d;s�be_WfEd�Z�Z�_Wjl�1_n�!¯ seo1_agOsM\;jl�l\�bM���kd;^�ikgOseikfD°J�1jlfEseikd;j9}�� �� ��x~�� � ~� ��������7© � {��� � q�_Ebe{�d��kikfRsM\;j1^�_Wj.sn}�� �� ��ì~�� �6~r¡��¾sM\;j1o �� � � ©

Page 12: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

ÞRÒ!$Ìr01�*KM5E�6�l5WK�=?��57Ò!$DC7$�Ìr01Ó�Ó�$;�*G�$;GºKM=?Ï Ó�0 =JG�Ï =?< $;���6P,0.< $aÒ*�'<1$�P,$�$��¾ÌWÒ!01K7$���K7057Ò*�65;-�� ��� � � ��Wþ���ÿ�� � �Àü7ý � �� �.ü�� �Wþ¾ÿ������ � �»K7$�$�Î!CE$�<>=?0 (*K�G!=?KEÌr(*KEK7=U01� �n-*57Ò!$]<.� C7=J�6�*Ì�$06Ö�5EÒ!$�0 (!57Î!(�5WK@Ña=UI?I��6IJK703P,$�ÌrI?01K7$Ù5E0 ��P,$;Ì;�6(*K7$�57Ò!$�$6V,$�Ìn5E=U<1$�Ïl�6=?� 0 Ö�57Ò*$�KM=?Ï Ó�0 =JG=?KC701(!Ï Ò!I?Ð��]0.<1$�CR=U5EK�(9KM$�ÖÝ(!IvCW�6�*Ï $ A�áh�#Î9�6C757=JÌr(!IJ�6C�->57Ò!=JK:KM=?Ï Ó�0 =JG3Ò*� Ka57Ò!$]Î*C701Î9$;CM5E=U$�K�»� � ��� � � �1� � � - �ÝP �a57Ò!$�KM$�Ìr0 �9G#G�$�CE=U<.�657=?< $�=JK��Ó��.æ�=?ÓÙ(!Ó �.5 � � �1-9� �*G �»Ì �R57Ò*$$4V�$;Ìn5E=U<1$�Ï1� =U�¾=?K�Ì�IU0lKM$:5E0B�1A

BD�!$�06Ö:57Ò!$#Î,065E$��l57=J�6IaÎ!CE0 P*IU$;Ó�K�Ña=U57Òx(9KM=?�!ϼKMÐ>Ó�Ó�$r5EC7=JÌ�KM=?Ï Ó�0 =JG!K�=JK�57Ò9�.5�57Ò*$$�CEC701C4K7(!C7Ö»� Ìr$@Ì�� �ºP,$,5���ü����,�.5a�!$;� C457Ò!$@0 CE=?Ï =?�%A�î!0 C45EÒ!=?KRCE$;�1KM01�º=U5a=?KRÏ10l0�G�570��'< 01=?G=U�!=U57=J�6I?=?ë�=?�!ϾÑa=U57Òå< $;C7Ð#K7Ó��6I?I�Ñ4$�=?Ï Òl5WK�Ac24$;Ì;�6(*K7$Ù0 Ö�57Ò!$�KE�.5E(!CE�657=?0 �Á0 Ö�57Ò*$�K7=?Ï Ó�0 =JG!K;-57Ò!$�$�CEC701CDK7(!C7Ö»� Ìr$�=JKD� I?K70 �9�65DÖ»� C:ÖÝC701Ó 5EÒ!$�0 CE=UÏ1=U�/A �DG!G�=?�!Ï3�¾KMÓ�� IUI®I?=U�!$��6CD57$;C7Ó 5E057Ò!$]K7=UÏ1Ó�0 =JG¾Ì;�6��KM01Ó�$�57=?Ó�$;KRÒ!$;IUÎ#�'< 01=?G�57Ò!$ �,�.5�CE$�Ï =?0 �9K�A����� � �����& �"03�4 � �2')4<% � �!.�*C%1 áh��ÌrIJ� KEK7=¦â9Ì;�.5E=U01�]Î!CE0 P!I?$�Ó�K;-;5W�6CEÏ $�5v<.�6I?(!$;K®� C7$�5eÐlÎ*=?Ì;�6I?IUÐ�P!=?�*� C7Ð �Ý$1A Ï9A 3 L � - 7 �Q=P�nA'&)01Ó�LÓ�01�ºÑa=JK7G!0 Ó Ó�=UÏ1Òl5RK7$�$;Óõ570�KM(*Ï Ï $�Ke5)57Ò*�65)5EÒ!$:5E� C7Ï1$r5)<.� IU(!$�K4P9$�K7$r5a�.5457Ò!$@<.�6I?(!$:06Ö57Ò!$]K7=UÏ1Ó�0 =JG R Ka�1KMÐ>Ó�Î�5E0657$�K�A!Ø0.Ñ)$;< $;C;-157Ò!=JKaÒ*�1KaKM$;< $;CE� I�G!CE�'ÑaP*�1ÌWÍ�K�A

îv=?CWKe5�-;=?�*KM5E�6P*=UI?=¦5E=U$�K®Ì��6�ÙC7$�KM(!IU5;A.ÞRÒ!$)57CW�6=?�!=?�!ÏÎ*C70�Ìr$�K7K/Ña=UI?Il57CEÐ@570:G�C7=?< $�57Ò!$40 (!57Î!(�5� KaÌrI?01K7$@� KRÎ,01KEKM=?P!I?$D5E0�57Ò!$:5W�6CEÏ $r5a<.�6I?(!$�K�-�ÑaÒ!=JÌWÒ#Ì�� �¾01�!IUкP9$�� ÌWÒ*=U$;< $;G3� K7Ð>Ó�Î�570 57=ULÌ��6I?I?Ð A �DK)�ÙC7$�KM(*I¦5�- 5EÒ!$Ñ4$�=?Ï Òl5EK �Ý01(�57Î!(!5R� �*G�$�<1$���Ò!=JG!G�$;� ��� C7$DG�C7=?< $;��570�I?� C7Ï1$�C�� �*GI?� C7Ï1$�C�<'� IU(*$;K�ÑaÒ!$�CE$)57Ò!$�K7=UÏ1Ó�0 =JGÙG�$;C7=?<.�.57=?< $)=?K�Ì�IU0lKM$)570@ë;$�CE0*A'ÞRÒ!$R< $�CEÐ�IJ�6CEÏ $�Ñ4$�=?Ï Òl5WK=U�*Ì�C7$�� K7$�5EÒ!$DÏ1CE�1G�=?$��l5EK;- Ò!0.Ñ4$�<1$�C�-65EÒ!$;K7$DÏ1CE�1G�=?$��l5EK)�6CE$�57Ò*$��ºÓ�(!IU57=?Î!IU=?$;G¾P>Ð��6�º$�æ�Î90 L�!$��l57=J�6I?I?Ð�K7Ó��6I?I9K7=UÏ1Ó�01=?G�G!$�CE=U<.�.5E=U<1$ �Ý$�æ!Ìr$�Î!5�ÑaÒ!$��¾�@5eÑa=?KM57=?�!Ï�5E$�CEÓ�F�=JK�� G*G�$;G�5E0]57Ò*$KM=?Ï Ó�0 =JG �4Î!CE0�G�(*Ì�=U�!Ï���Ñ4$�=?Ï Òl5�(!Î�G!�.5E$]ÌrI?01K7$@570�ë;$�CE0*A"�K��C7$�KM(!IU5;-!57Ò!$�Ñ4$�=?Ï Òl5EK�Ó��'ÐP9$�Ìr0 Ó�$]KM57(*ÌWÍ�A?>$;Ì�0 �*Gc-,ÑaÒ!$;�#57Ò!$�0 (!57Î!(�5WKDKE�.5E(!CW�.57$1-*5EÒ!$Ù�*$r5eÑ40 CE;Ï1=U<1$;K��*0º=?�*G�=JÌ��.5E=U01�Á06Ö4Ìr0 �!Lâ9G�$��9Ìr$]I?$�<1$�I[A*äxÒ!$;�Á�6�#=?�!Î!(�5Î9�.5M5E$�CE��Ö»�6I?I?K�!$;� C��G�$�Ìr=JKM=?0 ��P,0 (!�9G!�6CEÐ�5EÒ!$]0 (!57Î!(�5ÌrIJ� KEK�=?K�(*�*Ìr$;CM5W�6=?�%A�áeG�$;� IUI?Ðz5EÒ!=JK�KMÒ*0 (!IJGTP9$#CE$��*$�Ìn5E$;GT=U�T5EÒ!$#�!$r5eÑ40 CEÍzP>Ðã� �ã01(�5MLÎ!(�5@<'� IU(*$]57Ò9�.5D=JKD=U�ÁP,$r5eÑ4$�$;�#57Ò*$]5eÑ40�Î90lK7K7=?P!IU$Ù5E� C7Ï1$r5<.�6I?(!$�K�-,=ÀA $ A,�!0 5D�!$��6C$;=¦5EÒ!$�C� K7ÐlÓ�Î�5E065E$ A'Ø�0.Ñ4$�< $;C;-�I?� C7Ï1$�Ñ)$;=UÏ1Òl5EK/57$��9G�570ÖÝ0 CWÌr$4�6I?I101(�57Î!(!5EKv57057Ò!$)5E� =UIJK/0 Ö�57Ò!$RK7=UÏ LÓ�01=?G�CE$�Ïl�6CWG�I?$;KEK®06Ö,57Ò!$D(!�*Ìr$;CM5W�6=?�l5eÐ A ÞRÒ>(*K;-65EÒ!$��!$�5eÑ)01C7Í�Ó��'Ð�Î!CE$;G�=JÌn5)�@ÑaCE0 �*Ï]ÌrIJ� KEKÑa=¦5EÒ!0 (�5:Ï =?<>=U�!Ͼ�6�>о=?�*G�=JÌ��657=?0 �Á0 Ö�=¦5WK�I?0.ÑÚÌr01��â9G�$;�*Ìr$]=?�#5EÒ!$ÙCE$;K7(!IU5;A�"/�6CEÏ $�Ñ4$�=?Ï Òl5WK57Ò*�65DK7�657(!CW�.5E$:57Ò!$]�*0>G!$;KaÓ��6Í1$@=¦5=?Ó�Î90lK7K7=?P!IU$@5E0ºG�=/V,$;C7$;�l57=J�.57$�P,$r5eÑ4$�$;�35eÐlÎ*=?Ì;�6I/� �*G�!0 �l5eÐ>Î!=JÌ��6Ic$�æ!�6Ó�Î!I?$;K;A��KM01IU(�5E=U01�ç5E0ú57Ò!$�KM$#Î*C701P!IU$;Ó�K�=?K�570¼KM$�5�57Ò!$Á5E�6CEÏ $�5�<'� IU(*$;K�5E0åP,$ÁÑa=U57Ò!=?�ì57Ò*$CE� �!Ï $�0 Öc57Ò!$:KM=?Ï Ó�01=?Gc-1CE�657Ò!$;C�5EÒ*�6�º�65�5EÒ!$:�1KMÐ>Ó�Î�5E0657=JÌa<.�6I?(!$�K�A!&4�6CE$�ÓÙ(9Ke54P9$D5E� Í $��/-Ò!0.Ñ)$;< $;C;->5E0�=U�9KM(!CE$�5EÒ*�.5�5EÒ!$]�!0�G�$Ù=?K�!065CE$;KM57CE=JÌn57$�G3570�01�!I?Ð�57Ò*$ÙI?=U�!$��6C�Î9�6C7506Ö®57Ò*$KM=?Ï Ó�0 =JGcAQ?>$�5M57=?�!Ï:57Ò*$�5W�6CEÏ $r5®<.�6I?(!$;Kv5E05EÒ!$RÎ901=U�l5�06Ö!5EÒ!$RÓ��.æ�=UÓ�(!ÓÚK7$;Ìr01�*GÙG!$�CE=U<.�.5E=U<1$0 �ú57Ò*$3K7=UÏ1Ó�0 =JG =JKÙ57Ò!$¾P9$�Ke5�Ñ4�'Ð 570#5W�6Í1$º� G�<.� �15W�6Ï1$�06Öa57Ò*$º�!01�!IU=?�!$��6CE=¦5eÐåÑa=¦5EÒ!0 (!5K7�657(!CW�.5E=U�!Ï 57Ò*$ÁKM=?Ï Ó�0 =JGcA�ÞRÒ!=JK�=JK��6�!0 57Ò!$;C�C7$�� K70 �ç57Ò!$ÁK7=UÏ1Ó�0 =JGç=?�xîv=?Ï (*C7$ � PT=JK��Ï 0>0�G#ÌWÒ!0 =JÌr$1A*áO5Ò*�1K�Ó��.æ�=?ÓÙ(!Ó�K7$;Ì�0 �*G�G�$;C7=?<.�.57=?< $��.5 � �]ÑaÒ*=?ÌWÒÁÌ�0 CEC7$�KMÎ,0 �*G¾5E0�57Ò*$P!=U�9�6CEÐ�5E� C7Ï1$r5a<.�6I?(!$;K45eÐ>Î!=JÌ�� Ic=U�#ÌrIJ� KEK7=¦â9Ì;�.5E=U01�ºÎ!CE0 P!I?$�Ó�K;A

� �2')4&%(�� &)Ò!0>0lKM$R5W�6CEÏ $r5�<.� IU(!$�K��.5�57Ò!$�Î,0 =?�l5�06Ö,57Ò*$�Ó��.æ�=UÓ�(!ÓõKM$�Ìr0 �9G�G!$�CE=U<.�.5E=U<1$a0 ��57Ò*$K7=UÏ1Ó�0 =JG¾K70�� K4570��'<10 =JGºKE�.57(*CE�657=?�!ÏÙ5EÒ!$�0 (!57Î!(�5D(!�!=U5EK;A

� t s[��ikgOseikjl^Ôse_EbeZ ikgõ\�ghZ]\;�k�ì�kikjl_W\�bõse_EbeZ \;�l�1_W� sedÔseo1_�jld6�1_íd�� seql� sn}ú_�© ^1©� �� ��åsM\;j1o �� � �� � ©

Page 13: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

����� � 3�" ��"0�!.�"�� "03�4 ���C%���%("�4&�;�� ÞRÒ!$�Ke5W�6C757=?�!Ϻ<.�6I?(!$�KD0 Ö�5EÒ!$�Ñ4$�=?Ï Òl5WK:Ì;�6� Ò*�'< $��¾K7=UÏ1�!=Uâ9Ì�� �15@$4V�$;Ìr5�0 �Á5EÒ!$�57CW�6=?�!=?�!ÏÎ!C70�Ì�$;KEK�A®äå$�=?Ï Òl5EK�K7Ò!0 (!IJGzP,$ÁÌWÒ!0lKM$;�zCE� �*G�01Ó�I?мP!(�5�=?�TK7(*ÌWÒT� ÑR�'Ðå57Ò*�65�57Ò!$#K7=UÏ LÓ�01=?Gº=?K4Î!CE=?Ó�� C7=?I?Ð��1Ìn57=?<.�.5E$;G�=?�º=U5EK4I?=U�*$;�6C4CE$�Ï =?0 �/A1áOÖ/Ñ4$�=?Ï Òl5WK)� C7$:�6I?I9<1$�CEÐ�IJ�6CEÏ $�57Ò!$;�57Ò!$�K7=UÏ1Ó�01=?G Ña=UI?I�KE�.57(*CE�657$�C7$�KM(!IU57=?�!Ï3=U�úK7Ó�� IUI�Ï1CE�1G�=?$��l5EK57Ò*�65�Ó�� Í $�IU$��6CE�!=?�!Ï3K7I?0.Ñ�AáOÖÑ)$;=UÏ1Òl5EK��6CE$º< $;C7ÐúK7Ó�� IUI)57Ò*$��TÏ CW� G�=?$��l5EKÙÑa=UI?Ia�6IJKM0 P9$�< $;C7ÐúK7Ó�� IUI[A�áh�15E$�CEÓ�$;G�=J�.5E$Ñ)$;=UÏ1Ò15WK457Ò*�65CW�6�!Ï1$D0.<1$�CR57Ò*$]KM=?Ï Ó�0 =JG R KRIU=?�!$��6C�CE$�Ï1=U01�ºÒ*�'<1$D5EÒ!$Ù�1G�<.�6�l5E� Ï $D57Ò*�65 � ���57Ò!$�Ï1CE�1G�=?$��l5EK�� C7$�IJ�6CEÏ $�$��!01(!Ï Òå5EÒ*�.5]I?$;� C7�!=?�!Ï#Ì�� �úÎ!C70�Ì�$�$;Gú� �*G�� E �@57Ò!$��!$�5eÑ)01C7ÍÑa=UI?I!I?$;�6CE��57Ò!$�I?=?�!$;� C�Î*�6C75�06Ö,57Ò!$�Ó�� Î!Î!=?�!Ï@P9$�ÖÝ0 CE$R57Ò!$aÓ�0 CE$�G�=UÕ�Ìr(!IU5��!0 �*IU=?�!$;� C�Î*�6C75;A�ÌWÒ!=?$�<>=?�!Ï]5EÒ!=?K4CE$;Ül(!=?C7$�K)Ì�0>0 CWG�=U�9�.57=?0 �ºP9$�5eÑ)$;$���5EÒ!$D57CW�6=?�!=U�*Ï�K7$r5a�!0 CEÓ��6I?=Uë��.5E=U01�%-57Ò!$ÌWÒ*0 =JÌr$�0 ÖcK7=?Ï Ó�0 =JGc- � �*G�57Ò!$DÌWÒ!0 =JÌr$�0 ÖcÑ)$;=UÏ1Ò15�=?�!=U57=J�6I?=Uë��.5E=U01�%A äå$KM5E�6C75�P>ÐÙCE$;Ül(!=?C7L=U�!Ï�5EÒ*�.5a5EÒ!$]G�=JKe5EC7=?P!(�5E=U01��06Ö/5EÒ!$�0 (!57Î!(�5WKa06Ö®$;�1ÌWÒ¾�*0>G!$�Ò*�'< $@��Ke5W�6�*G*�6CWG¾G�$;<>=?�657=?0 ���� �D06ÖR�6Î!Î!CE0'æ�=UÓ��657$�I?Ð � AcÞRÒ!=JK@=?K@� ÌWÒ*=U$;< $;G �.5:57Ò!$�=?�!Î!(�5�IJ�'Ð $�C:P>Ð#�!0 CEÓ��6I?=Uë;=U�*Ï�57Ò*$57CW�6=?�!=U�*Ï KM$�5�� K�G�$;KEÌrCE=?P9$�G¼$;� C7I?=U$;C;AvÞ/0Á01P�5E� =U�z� Ke5W�6�*G!� CEG¼G!$�<>=?�657=?0 �zÌ�IU0lKM$�5E0 �º�6557Ò!$@0 (!57Î!(�5a0 Ö%57Ò!$:â*CWKM54Ò!=JG!G�$;�¾IJ�'Ð $;C)Ñ4$ �e(*Ke5a�!$;$;G�570�(*KM$:5EÒ!$��6P,0.< $CE$;Ì�0 Ó�Ó�$��*G!$;GKM=?Ï Ó�0 =JG�5701Ï $�57Ò!$;C�Ña=U57Ò�5EÒ!$CE$;Ül(!=?C7$;Ó�$��l5�57Ò9�.5�5EÒ!$=?�!Î!(!5)5E0�57Ò!$:KM=?Ï Ó�01=?G�� I?K70]Ò*�'<1$��Ke5W�6�*G!� CEG�G�$;<>=?�657=?0 ������� � A �KEKM(!Ó�=?�!Ï�5EÒ!$Ù=?�!Î!(�5WK�- � , -*5E0º��(!�!=U5:� C7$�(*�*Ìr01C7CE$�IJ�.5E$;GÑa=¦5EÒ3<.� C7=J�6�*Ì�$�� ->57Ò*$]Ke5W�6�*G!� CEG¾G�$�<>=J�.5E=U01�¾0 Öv5EÒ!$�(!�!=U5EKaÑ4$�=?Ï Òl5E$;G3KM(!Ó Ña=UI?IcP,$

� ��� � J � F,! �� +��EF 5 �%� � �

ÞRÒl(9K�-'5E0:=?�*K7(!CE$�5EÒ*�.5®57Ò*$�� ��� � C7$R�6Î!Î*C70'æ�=?Ó��657$;IUÐ��457Ò!$RÑ4$�=?Ï Òl5EK®K7Ò!0 (*I?G�P9$RCW�6�*G!0 Ó�IUÐG�CE�'Ña�¾ÖÝCE0 Óí��G�=JKe5EC7=?P!(�5E=U01��Ña=¦5EÒ3Ó�$��6�3ë�$�CE0��6�*G3��Ke5W�6�*G*�6CWG¾G�$;<>=?�657=?0 �3Ï =?< $��3P>Ð��� ��� � +��7F �%��" �ÑaÒ!$�CE$��Ô=?K45EÒ!$��>(!ÓÙP,$�Ca0 Ö�=?�!Î!(!5EKR5E0�57Ò!$�(!�*=¦5�A

� 3�" ��"��!.0"�� "�3C4�� %1"04<�;�� �DK7K7(!Ó�=U�*Ï�5EÒ*�.5 C�1A)5EÒ!$@57CW�6=?�!=?�!Ï�KM$�5aÒ*� KRP,$�$;�3�*0 CEÓ�� IU=?ë�$�Gc-!�6�9GE A)5EÒ!$]K7=UÏ1Ó�01=?GºÖÝCE0 Óíî�=?Ï (!CE$ � P3Ò*� KRP,$�$;�3(9KM$�G5EÒ!$���Ñ4$�=?Ï Òl5EKRK7Ò!0 (!IJG¾P9$�CW�6�*G!0 Ó�IUÐ�G!CE�'Ña�ºÖÝC701Óí��G�=?KM57CE=?P!(�57=?0 � �Ý$1A Ï9A!(!�!=UÖÝ0 CEÓ&�Ña=U57Ò�Ó�$;� �¾ë;$�CE0�� �*G3Ke5W�6�*G!� CEG3G�$;<l=J�.5E=U01���� ��� � +��EF � � & �ÑaÒ!$;C7$���=JKR57Ò*$@Ö»�6��LO=U� �×5EÒ!$]�>(!Ó�P9$;Ca06Ö�Ìr0 �*�!$;Ìr57=?0 �*K)ÖÝ$�$�G�=?�!Ï#ÿ���� �D57Ò!$��*0>G!$ �nA

����� � �����& �"03�4 @ %1�2')3C"�3C4 ')�2��%( ÞRÒ!$�CE$Ù=JKD�65DI?$;�1Ke5:0 �!$�Ñ)$;IUIUL[Î*C7=?�*Ìr=?Î!I?$;G#Ó�$�57Ò!0�G��ÀG�$;KEÌrCE=?P9$�G#=U�åKM$�Ìn5E=U01� -!A E �4ÖÝ01CD$�Ke5E=¦LÓ��657=?�!Ï�57Ò!$@=JG�$;� I,I?$;� C7�*=U�!Ï�CE�657$ 0 A!Q �6�>Ð�0657Ò*$�CaK7ÌWÒ*$�Ó�$;KG�ÝÓ�0lKe5R06Ö%5EÒ!$�ÓíCE�657Ò!$;C)$;Ó�LÎ!=UCE=JÌ��6IU�vÒ*�'< $4P,$�$���Î!CE0 Î,01K7$;GÙ=U��57Ò*$4I?=U57$�CW�.5E(!CE$�5E0@� (�5701Ó��.57=JÌ�� IUI?Ð]� GP�e(*Ke5�5EÒ!$RIU$��6CE�!=?�!Ï

Page 14: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

CE�657$ AcQ#0lKe5@06Ö�57Ò!0lKM$�K7ÌWÒ*$�Ó�$;K:G�$;Ì�C7$�� K7$]5EÒ!$�IU$��6CE�!=U�*ϺCW�.57$�ÑaÒ!$;�Á57Ò*$�Ñ4$�=?Ï Òl5:< $;Ìr5701CèM01KEÌr=?I?I?�657$;KEê!-c� �*G =?�*Ì�C7$�� K7$�=U5]ÑaÒ!$;� 5EÒ!$�Ñ4$�=?Ï Òl5�< $;Ìr5701CDÖÝ01IUI?0.Ñ�K��3C7$;I?�657=?< $;IUÐ KM57$�� G�ÐG�=UCE$;Ìr57=?0 �%A'ÞRÒ!$4Ó��6=?�ÙÎ!CE0 P!I?$�Ó Ña=U57Ò�57Ò!$�KM$)Ó�$r57Ò*0>G*Kv=JKv57Ò*�65v5EÒ!$�Ð]� C7$)�!065��6Î*Î!C701Î!CE=?�657$ÖÝ0 C�Ke5E0�ÌWÒ*� KM57=JÌÙÏ1CE�1G�=?$��l5D01C:01��LOIU=?�!$�IU$��6CE�!=U�*ϾP,$;Ì;�6(*K7$Ù5EÒ!$�Ñ)$;=UÏ1Òl5:<1$;Ìn5E0 C �*(*Ìr57(*�657$;K�6I?I�57Ò!$@5E=UÓ�$ A

2)$;Ð 01�*G#ÌWÒ!0>01K7=U�*Ͼ�ºK7=?�!Ï I?$�Ï1IU01P*�6IvIU$��6CE�!=?�!ϺCW�.5E$ -,=¦5@=?K@Ì�IU$��6CD57Ò*�65:Î!=JÌWÍ>=U�*Ͼ�3G�=¦Ö×LÖÝ$�CE$��l5�IU$��6CE�!=?�!ϼCE�657$ 0 ,�ÖÝ0 C�$�� ÌWÒìÑ)$;=UÏ1Òl5ºÌ��6�ì=?Ó�Î!C70.<1$357Ò!$ Ì�0 �>< $;C7Ï1$��*Ì�$ A<��Ñ4$�I?IULÎ!C7=?�*Ì�=UÎ!I?$;G�ÑR�'Ð:0 Ö,G�01=U�*Ï5EÒ!=?K;-.P*�1KM$�GÙ01��Ìr01Ó�Î!(�57=?�!Ï�K7$;Ìr01�*GÙG!$�CE=U<.�.5E=U<1$;K;-'=?K®G�$�K7Ì�C7=?P9$�G=U� K7$;Ìr57=?0 � -*A/�1A9ÞRÒ!$�Ó��6=?�#Î!Ò!=?IU0lKM01Î!Ò>о=JK�5E0ºÓ��6Í1$ÙK7(!C7$]57Ò*�65:�6I?I%5EÒ!$�Ñ)$;=UÏ1Ò15WK�=?�#57Ò*$�!$r5eÑ40 CEÍ�Ìr01�l<1$�CEÏ $DC701(!Ï Ò!I?к�6545EÒ!$]KE�6Ó�$�KMÎ,$�$�GcA

Û$;Î9$;�*G�=?�!Ï@(!Î901�Ù5EÒ!$�Ìr(*C7<.�.5E(!CE$)0 Ö!57Ò!$a$;C7CE0 C®K7(!CMÖ»�1Ìr$1-'K70 Ó�$4Ñ)$;=UÏ1Òl5EK�Ó��'Ð�C7$�Ü1(*=UCE$�aKMÓ��6I?I6I?$;� C7�!=?�!ÏRCW�.57$�=?��0 CWG�$;C�570��'< 01=?G:G�=?< $;C7Ï1$��*Ì�$ -nÑaÒ!=?IU$�0657Ò*$�CWK�Ó��'ÐDC7$�Ül(!=UCE$��RI?� C7Ï1$IU$��6CE�!=U�*ÏCW�.57$4=?��01CEG�$;C/5E0�Ìr0 �><1$�CEÏ $��65®�DC7$�� K70 �*� P!I?$)K7Î9$;$;GcA.24$;Ì;�6(*K7$)0 Ö!57Ò*=?K;-.IU$��6CE�!=?�!ÏCE�657$;K�=U��57Ò!$:IU0.Ñ4$�C�I?�'Ð1$�CWK�KMÒ*0 (!IJG�Ï $;�!$�CW�6I?I?ÐÙP,$IJ�6CEÏ $�C�57Ò*� ��=U�º57Ò!$DÒ!=?Ï Ò!$;C�IJ�'Ð $�CWK �ÀKM$;$îv=?Ï (*C7$ E � �nA9ÞRÒ!=?KDÌr01C7CE$;Ìr5EK4ÖÝ0 C�57Ò!$�Ö»�1Ìn5a5EÒ*�.5=?�#Ó�01KM5�!$;(!CE� Ic�!$r5:�6CWÌWÒ!=¦5E$;Ìr57(!CE$;K;-�57Ò*$KM$�Ìr0 �9G�G�$�CE=U<.�657=?< $0 Ö�57Ò!$@Ìr0lKe5�ÖÝ(*�*Ìn5E=U01�ºÑa=U57Ò�CE$;K7Î9$�Ìn5)570ÙÑ4$�=?Ï Òl5WK�=?��57Ò!$:I?0.Ñ)$;C�I?�'Ð1$�CWK=?K:Ï $;�!$�CW�6I?I?Ð3K7Ó��6I?IU$;C5EÒ*�6� 57Ò*�65:0 Ö�57Ò!$�Ò!=?Ï Ò!$;CDIJ�'Ð $;CEK;A9ÞRÒ*$�CW�.5E=U01�*�6I?$]ÖÝ01C5EÒ!$�� P90.<1$Ò!$�(!CE=JKe5E=?Ì;KvÑa=?I?I�P9$aG�=JK7Ì�(*KEKM$�G]=?��Ó�0 CE$)G!$r5E� =UI�=?��I?�657$�C®K7$;Ìn5E=U01�*K®�6I?0 �!Ï:Ña=U57Ò�KM(!Ï1Ï $�Ke5E=U01�*KÖÝ0 C�Ò!0.Ñ 5E0�ÌWÒ!0>01K7$@57Ò!$Ù�1Ìn5E(*�6Iv<'� IU(*$�06Ö®57Ò!$ÙIU$��6CE�!=U�*Ï�CE�657$@ÖÝ0 C57Ò!$�G�=WV�$�CE$��l5Ñ4$�=?Ï Òl5WK�»K7$�$]K7$;Ìn5E=U01� -*A/���nA

áOÖKMÒ*� C7$�GúÑ)$;=UÏ1Òl5EKÙ� C7$�(9KM$�GzKM(9ÌWÒç� KÙ=U�¼5E=UÓ�$�LOG�$;I?�'Ðå�!$;(!CE� I��!$�5eÑ)01C7Í�K&�ÀÞaÛ � � �� � E �v01CDÌ�0 �>< 01IU(�5E=U01�*�6I/�!$r5eÑ40 CEÍ�K � E � �O-!5EÒ!$ÙI?$;� C7�!=?�!Ï�CW�.5E$ÙK7Ò!0 (!IJG#P,$ÙÎ!CE0 Î,0 C757=?0 �9�6Ic5E057Ò!$�KEÜl(*�6CE$�CE0>065�06Ö)57Ò!$��>(!Ó�P9$;C�06ÖaÌr01�!�!$;Ìr57=?0 �*K�K7Ò*� C7=?�!Ͼ57Ò*�65@Ñ4$�=?Ï Òl5�-cP9$�Ì��6(9KM$�Ñ4$Íl�*0.Ñ�57Ò9�.5a57Ò*$�Ï CW� G�=?$��l5EKR� C7$@��KM(*Ó�06Ö®Ó�0 CE$rLO0 C7L[I?$;KEK�=?�*G�$;Î9$;�*G�$;�15a5E$�CEÓ�K�A

���*��!.0"�� %A���C%A@ %1�2')3C"�3C4 � �B%1% 9 � Ï1=U<1$@$;� ÌWÒ¾Ñ)$;=UÏ1Òl5R=U5EK�0.Ña�¾I?$;� C7�!=?�!Ï�CW�.57$� I?$;� C7�*=U�!Ï�CE�657$�K4K7Ò!01(!I?GºP9$@Î!CE0 Î,0 C757=?0 �9�6I9570�57Ò!$�KEÜ1(9�6CE$CE0l0 540 Ö%57Ò*$

�>(!Ó�P9$;Ca06Ö®=U�*Î!(�5EKa570�57Ò!$�(*�!=¦5� Ñ4$�=?Ï Òl5WK@=U�¼I?0.Ñ)$;C�I?�'Ð1$�CWK�KMÒ*0 (!IJG 5eÐlÎ*=?Ì;�6I?IUÐ P9$�IJ�6CEÏ $;C:57Ò*� �¼=U�å57Ò*$

Ò*=UÏ1Ò!$�CaIJ�'Ð $;CEK

B57Ò!$;CR57CE=?ÌWÍ�K)ÖÝ0 Ca=?Ó�Î!C70.<>=?�!Ï�57Ò!$]Ì�0 �>< $;C7Ï1$��*Ì�$=?�*ÌrI?(*G�$ C� � �%(3;��* Q#01Ó�$��l57(!Ó

��� ��.�7 � �<� 0 � ��(�� +���

7�� ��� �/.%�6Ì��6��=?�*ÌrCE$;�1KM$RKMÎ,$�$�GÙÑaÒ*$���57Ò!$�Ì�01KM5�K7(!C7Ö»� Ì�$)=JK®Ò!=?Ï Ò!I?Ð]�!01�*KMÎ*Ò!$�CE=?Ì;�6I>P,$;Ì�� (*K7$4=U5�G!�6Ó�Î*K57Ò!$#K7=Uë;$¾0 Ö�57Ò!$#KM57$;Î*K��6I?0 �!ÏåG�=UCE$;Ìr57=?0 �*K�0 ÖÒ!=?Ï ÒãÌ�(!CE<'�657(!CE$¾57Ò>(*K�Ð>=U$;I?G�=?�!Ïú� IJ�6CEÏ $;C$4V�$;Ìn5E=U<1$ÙI?$;� C7�!=?�!Ï�CW�.5E$Ù�6I?0 �*Ï�57Ò!$�G�=?C7$�Ìn5E=U01�*K�0 Ö�IU0.ÑÚÌ�(!C7<.�657(!CE$ � � � ����çG!$��!0 57$;K57Ò*$Ke5EC7$;�!Ï65EÒ�0 Ö95EÒ!$aÓ�0 Ó�$��l57(*ÓÚ57$;C7Ó&�rA6áO5�Ò*� K�P9$;$���ÌrIJ�6=?Ó�$;GÙ57Ò*�65�Ó�01Ó�$��l57(!Ó Ï $;�!$�CW�6I?I?ÐÒ!$�I?Î*K�Ó�01C7$�=?�ÁP*�65EÌWÒ#Ó�0�G�$�57Ò9�6�#=?�ÁKe5E0>ÌWÒ9� KM57=JÌ@Ó�0�G�$1-9P*(�5�!0ºK7Ð�Ke5E$�Ó��.5E=?ÌÙKe5E(*G�Ð306Ö57Ò!=JK��6CE$@Í>�!0.Ña��5E0�57Ò!$]� (�57Ò!01CEK;A

Page 15: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

8 9 ���C��" #<% .�%(�!'-3�"03�4 ')�2��%1 QÁ� �>Ð �6(�5EÒ!0 CWK;-c=U�*Ì�IU(9G�=U�*Ï�?>0 Ó�Î,0 I?=U�*K7Í>ÐÁ$r5Ù� IÀA � � ( �O-ÛD� C7Í1$���� Q#0>0>G!Ð � - �[-<?>(!5M5701� � � * �[-�Q#(*CE�65E� $r5��6I[A � E * �aÒ*�'< $3Î!CE0 Î,01K7$;G¼CE(!I?$;K�ÖÝ01C�6(�5E0 Ó��.5E=?Ì;�6I?IUÐ�� G!� Î�57=?�!Ï�5EÒ!$I?$;� C7�*=U�!Ï�CW�.5E$;K �»K7$�$D� I?K70 � � & �*�nA>ÞRÒ!$�KM$�CE(!I?$;K)Ìr0 �l5EC701I!57Ò*$KMÎ,$�$�G¾0 Ö�Ì�0 �>< $;C7Ï1$��*Ì�$P>Ð�=?�*ÌrCE$;�1KM=?�!Ï�0 C�G�$�ÌrCE$;� K7=?�!ÏÙ5EÒ!$@IU$��6CE�!=?�!Ï�CE�657$@P*� K7$;Gº0 �357Ò*$$�CEC701C;A

ä $�� KEKM(!Ó�$�5EÒ!$]ÖÝ01IUI?0.Ña=?�!Ï�Ö»� Ìr5EKÖÝ0 C:��IU$��6CE�!=?�!Ï�CW�.57$�� G!� Î�5E�657=?0 �Á�6I?Ï 01C7=U57Ò*ÓBC�� ���57Ò!$�K7Ó��6I?IU$�Ke5%$;=UÏ1$��><.�6I?(!$®06Ö>57Ò*$�Ø�$�K7K7=?� � �»K7$�$<A�Ü,A8� E (Q�%�,=JK%K7(�Õ�Ìr=?$��l57I?Ð@KMÓ��6I?I?$�C�57Ò9�6�@57Ò*$KM$�Ìr0 �9G#KMÓ��6I?I?$;KM5$�=?Ï $;�l<.� IU(!$��6�*G � E �a57Ò!$;C7$�ÖÝ0 CE$Ù�.Ö×5E$�CD��I?� C7Ï1$]�>(!ÓÙP,$�CD06Ö�=U57$�CW�.5E=U01�*K;-57Ò!$ Î*�6CW�6Ó�$r5E$�C�<1$;Ìn5E0 C � �/.%�ºÑa=UI?ID� Î!Î!CE01� ÌWÒã57Ò*$ Ó�=U�*=UÓ�(!Ó ÖÝC701Ó 5EÒ!$ G�=?C7$�Ìn5E=U01�x06Ö57Ò!$3Ó�=U�!=?ÓÙ(*Ó $�=?Ï $��><1$;Ìn5E0 C]0 Öa57Ò!$�Ø�$�K7K7=J�6� �»K7$�$BA�Ü,A8� E (X�n-®î�=UÏ1(!C7$ "X�rA���*G�$�C�57Ò*$;K7$Ìr0 �9G�=¦5E=U01�*K�57Ò!$:$�<10 I?(�57=?0 ��0 Ö%57Ò*$D$�Ke5E=UÓ��.5E$;G�Î9�6CW�6Ó�$r5E$�C)Ì;�6�¾P9$D57Ò!01(!Ï Òl5406Öv�1K)�Ù0 �!$�LG�=UÓ�$;�*KM=?0 �9�6I�Î!C70�Ì�$;KEK�6�9G�57Ò!$�Ó�=U�!=?ÓÙ(*Ó�$�=?Ï $��><1$;Ìn5E0 C # Ì�� �ÁP9$��6Î*Î!C70'æ�=?Ó��.57$�G �×ÖÝ0 C��IJ�6CEÏ $:�>(!ÓÙP,$�C06Öv=U57$;CE�657=?0 �*KDC!K7$�$�î�=UÏ1(!C7$ " �4PlÐ

# ��� � ������ ���

� ������ �P

ÑaÒ!$�CE$�� �:G�$;�!065E$;KR5EÒ!$���F:�!01C7Ó�A!Ø$��*Ì�$�Ñ)$�Ì�� �3�1G�0 Î!5��Î*C70X�e$;Ìn5E=U01�� ��� # + � �

��� � ���� � ���� � �

570¾57Ò!$��6Î*Î!C70'æ�=?Ó��.57$�G�Ó�=U�!=?ÓÙ(*Ó A�=?Ï $;�l<1$;Ìr570 C # �1K:�¾0 �*$�G�=?Ó�$��*K7=U01�*�6I®Ó�$;�1KM(*C7$�06Ö57Ò!$G!=?KM5E� �*Ìr$R5E0�57Ò!$Ó�=?�!=?ÓÙ(!Ó�AßÞRÒ!=JK�G�=JKM5E�6�9Ìr$�Ì�� ��P9$�(9KM$�G�570]Ìr01�l57CE0 I>57Ò*$�IU$��6CE�!=?�!ÏCE�657$ �×ÖÝ01C�G�$r5W�6=?I?K�K7$�$ � E * �U�

� �/.C7 � �<� � �/.�7 � � F10 ( � ��(��� �%� (X�

' �/.�7 �D� � � � F �Q� ' �/.%��7 �� ��(��� � ��� � � ��� �%� * �

0 �/.�7 �D� � 0 �/.%��7�� 0 ��.%� ����� ' �/.�7 � ��� F10 �/.%�;� �%� - �ÑaÒ!$�CE$��ÙÌr0 �l5EC701I?K�57Ò!$@I?$;� Í�K7=?ë�$:06Ö/5EÒ!$@�'< $�CW�6Ï1$ -�� ��å� C7$:Ìr0 �9Ke5W�6�l5EK4�6�*G ' =JK)(9KM$�G¾�1K�6(�æ�=?IU=J�6CEÐ�<.�6CE=?� P!IU$:5E0�Ì;�6IJÌr(!IJ�.5E$:57Ò!$�I?$;� Ílо�'< $�CW�6Ï1$0 Ö/57Ò*$�Ï CW� G�=?$��l5 ���

� � A��065E$º5EÒ*�.5�5EÒ!=JK�KM$�5�0 ÖCE(!IU$�K�=?K�$�� K7Ðú570¼Ìr01Ó�Î*(�57$#�6�9GçKM57CW�6=?Ï Òl5MÖÝ01C7ÑR�6CWGå570ú=?Ó�L

Î!IU$;Ó�$��l5;Acäå$�KM=?Ó�Î!IUÐ Ò*�'< $�570�Í $;$�Î 57CW� ÌWÍ#06Ö4�6�ú� G!G�=U57=?0 �9�6I®< $�Ìn5701C@=U� A�Ü,A8� � * �-C,57Ò*$�'< $�CW�6Ï1$;G:Ï CW� G!=U$;�15 ' A.ÞRÒ!$)�!0 CEÓ 06Ö�5EÒ!=?K�<1$;Ìn5E0 C/57Ò!$;��Ìr01�l57CE0 IJKc57Ò*$4K7=Uë;$�0 Ö�57Ò!$4IU$��6CE�!=?�!ÏCE�657$ �»K7$�$�A�Ü,A8� � - �%�rA ÞRÒ!$� IUÏ10 CE=¦5EÒ!ÓÚÖÝ01IUI?0.Ñ�K�5EÒ!$K7=UÓ�Î!I?$�=?�15E(!=U57=?0 � C1Ö»�6C4�'Ñ4�'Ð�ÖÝCE0 Ó�57Ò*$Ó�=?�!=?ÓÙ(!Ó �»I?� C7Ï1$ÙG�=JKe5W�6�*Ì�$ � �=U5DÎ!CE0�Ìr$�$�G!KD=U� P!=UÏ3Ke5E$�Î*K:�6�9G ÌrI?01K7$�5E0�57Ò*$�Ó�=?�!=UÓ�(!Ó=¦5� �!�!$;� I?K45EÒ!$�I?$;�6CE�!=?�!Ï�CE�657$ �×ÖÝ01CR57Ò!$;0 CE$r57=JÌ�� I%G�$r5W�6=?I?KKM$;$ � E * �U�rA����� �� 9 "��!. +��! �"0 "!C*C3��6��"0�<3C #� ��"04 �&" 9�# 3C"0�� ��IU57Ò!01(!Ï Ò¼Ó�01KM5�K7Ð�Ke5E$�Ó�K�(*K7$��!0�G�$;KÙP*� K7$;Gå01�zG�065ÙÎ*C70�G�(*Ìr5EK]� �*G¼K7=UÏ1Ó�01=?G*K�-%Ó�� �lÐ0657Ò*$�CR5eÐ>Î9$�KR06Ö�(!�!=U5EK �Ý01CaI?�'Ð1$�CWK#��Ì�� �¾P,$�(*K7$;GcA"� Ìr01Ó�Ó�01�3� I¦5E$�CE�*�.5E=U<1$D=JK457Ò*$:CW� G!=?� IP*� K7=?K�ÖÝ(!�*Ìr57=?0 ���ÀN�2Rî ���!$�5eÑ)01C7Í'�»K7$�$ � (�- E &!- "!- � E �U��áh�ºN�2Rîå�!$r5eÑ40 CEÍ�K�- 57Ò!$:G�065RÎ!CE0>G�L(*Ìn5�0 Ö95EÒ!$aÑ)$;=UÏ1Òl5�� �*G�=U�*Î!(�5�< $�Ìn5E0 C®=?K�C7$;Î!I?�1Ìr$�GÙÑa=U57Ò��GA�(*Ì�IU=JG�$��6��G�=JKe5W�6�*Ì�$4P,$r5eÑ4$�$;�

Page 16: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

L<M7N ¹��*¹  �d�j6�'_Ebe^;_WjlfW_�d�°/seo1_��>dn�a©>pR� beikjl^@seol_Rªlj>\��,gOsM\;^;_Dd�°%�k_n\�bej1i¦j1^�seol_\n�'_EbM\;^;_��>dn�Tikg\�qlq bedr½ ikZ]\�se_E�k�]d;jl_a� ikZ�_Wjlghikd�jl\;��sedn��\rbM�1g®seol_)Z�ikjlikZ:�1Z����R\;jl�Ùi�s�ikg�\D^;d.d6��\;q1q1bedn½ ikZ]\�|seikd;j�d;°�seo1_aZ�ikjlikZ:�1Z�_Wik^;_Wj6��\��¦�1_��1i�be_WfEseikd;j�d;°cseo1_ � _Wghghi¦\�j9©

57Ò!$D=U�!Î*(�54� �*G�Ñ)$;=UÏ1Òl5)�6�*G�5EÒ!$DK7=?Ï Ó�0 =JG�=?K�C7$;Î!IJ� Ìr$�G�P>Ð�� ��$ræ�Î901�!$��l5E=?� IÀAlÞRÒ!$�0 (!57Î!(�5� Ìn5E=U<>=U5eÐ�=JK�Ìr01Ó�Î*(�57$�Gc-!$ A Ï*A>ÖÝ01Ca0 �!$@01(�57Î!(!5;-*�1K

� � � � � J , Kv+

� , $ræ�Î� F �E � F, � � F � , � F�� ÑaÒ!$�CE$ � , ��� , ��=JK�5EÒ!$Ó�$��6���»KM5E� �*G!� CEG�G�$;<l=J�.5E=U01� �®0 Öc57Ò!$ �[L[57Òº8@� (*K7K7=J�6�%AlÞRÒ!$;K7$�(!�!=U5EKÌ��6� C7$;Î!I?�1Ìr$�0 CDÌ�0>$ræ�=?KM5DÑa=U57Ò 57Ò!$�KM5E�6�9G!�6CWG#(!�!=U5EK@� �*G�57Ò*$�ÐÁ�6CE$](*K7(*� IUI?Ð357CW�6=?�!$;G#P>ÐÌr0 Ó�P!=?�*�.5E=U01�30 Ö�Ï CW� G�=?$��l5G�$�K7Ì�$��l5 �×ÖÝ01C0 (!57Î!(�5:(!�!=U5EK �a�6�9G�(!�*K7(!Î9$;C7<>=JKM$�G#ÌrI?(*KM57$�CE=?�!ÏÖÝ0 C�G�$�57$�CEÓ�=U�*=U�!Ï�5EÒ!$�Ó�$;� �*Ka�6�*G¾Ña=JG>57Ò*K�06Ö�57Ò!$]N�2Rî¼(*�!=¦5WK�A

���!I?=UÍ1$�K7=?Ï Ó�0 =JG!�6I�(!�!=U5EK@ÑaÒ*=?ÌWÒåÌ��6�åÌr0.<1$�C�5EÒ!$�$��l57=?CE$�K7Î*� Ì�$ -��¾K7=?�!Ï I?$�N24îì(*�!=¦5Ìr0.< $;CEK:01�!IUÐå�#KMÓ�� IUI)IU0�Ì�� I�C7$;Ï =?0 �ú06Ö457Ò*$�=U�*Î!(�5�K7Î*� Ì�$ A/ÞRÒ*=?K]Ì;�6�úP,$º�6�z�1G�<.�6�l5E� Ï $P9$�Ì��6(9KM$�I?$;�6CE�!=?�!Ï#Ì�� � P,$�Ö»� KM57$;C;A%N�2Rî (!�!=U5EK�Ó��'Ð � I?K70ºÖÝ01C7Ó÷��P9$�5M57$;CÙK7$r5�06ÖRP*� K7=JKÖÝ(!�*Ìn5E=U01�*K:5703Ó�0�G�$�I�57Ò*$�=?�!Î!(!5�K7Î*�1Ìr$�57Ò*� � K7=UÏ1Ó�01=?G (!�!=U5EK;-%�6IU57Ò!01(!Ï ÒÁ57Ò!=JK@=?K:Ò!=?Ï Ò*IUÐÎ!C701P!I?$�ÓõG�$�Î,$��*G!$��l5;A*BD��57Ò*$D�!$;Ï1�657=?< $�K7=?G�$1- 5EÒ!$I?0�Ì�� IU=U5eÐ�Î*C701Î9$;CM5eÐ�06Ö/N�2Rî®K)Ó��'Ð�P,$�ÁG�=JK7�1G�<.�6�l5E� Ï $�Î*� CM5E=?Ì�(!I?� C7I?Ð =?�zÒ!=?Ï ÒzG!=UÓ�$��9KM=?0 �*� I4K7Î*�1Ìr$;K]P9$�Ì��6(9KM$�Ó��'Ðå(!�!=U5EK��6CE$�!$�$�G�$;Gç570úÌr0.<1$�C�57Ò!$#K7Î*� Ì�$;K;A®N24î®K��6CE$3Ó�01C7$3�6Î!Î*C701Î!C7=J�.5E$º=?� �»IU0.Ñ G�=UÓ�$;�*KM=?0 �9�6IU�(!Î!Î,$�C�IJ�'Ð $;CEK4�6�*G�K7=UÏ1Ó�0 =JG!K4=U� �ÝÒ*=UÏ1Ò�G!=UÓ�$��9KM=?0 �*� I8�)I?0.Ñ)$;CRI?�'Ð1$�CWK�A� � Ç)Ä��)ò%Æ!ô�ò/ÄDÊ�ò Ç��#öxÆ!ó�ÈËMò/Ä)Å��xò��>Ê�ò/Ä)Å

���� 8 @B"0� ��.0% � ��%1�:'��áh�Ù5EÒ!=?K®K7$;Ìr57=?0 �ÙÑ4$4$ræ!�6Ó�=?�!$4K70 Ó�$406Ö957Ò!$)57Ò!$;0 CEÐ�P9$;Ò!=?�*G]5EÒ!$457CE=?ÌWÍ�K�Î!CE$;K7$��l57$�G�$;� C7I?=?$�C�Aä $�P,$�Ï1=U��=?�301�!$]G�=?Ó�$��*K7=U01�¾ÑaÒ*$�CE$D5EÒ!$](!Î�G!�657$�$;Ül(*�657=?0 �¾ÖÝ0 CaÏ1CE�1G�=U$;�l5aG!$;KEÌr$��l5�Ì�� �P9$�ÑaCE=¦5757$;���1K

����.�7 � �<� ���/.%� F 0�� �&� ���� � 5 � E � �

Page 17: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

ä $]Ñ40 (*I?G�I?=UÍ1$@570�Í>�!0.Ñ Ò!0.Ñ 57Ò*$]<.�6I?(!$]06Ö 0 �PV�$;Ìr5EKÌ�0 �>< $;C7Ï1$��*Ì�$:� �*G357Ò*$]IU$��6CE�!=?�!ÏKMÎ,$�$�GcA/î�=UÏ1(!CE$ &3=?IUI?(*KM57CW�.57$�K:57Ò!$�I?$;� C7�*=U�!Ï3P,$�Ò*�'<>=?0 C:ÖÝ0 C]K7$�<1$�CW�6I�G�=/V,$;C7$;�15ÙK7=?ë�$;K:0 Ö 0ÑaÒ!$��º57Ò!$DÑ)$;=UÏ1Ò15�� Ke5W�6C75EK)0 (�54=U�º57Ò!$D<>=?Ì�=U�!=U5eÐ�06Öv�ÙIU0�Ì;�6I*Ó�=?�!=?ÓÙ(!Ó�A>áh�º01�!$DG�=?Ó�$���LKM=?0 �%-v=¦5�=?KÙ$;� K7ÐÁ5E0 G�$�â*�!$�5EÒ!$º01Î�57=?Ó��6I�I?$;� C7�!=?�!Ï#CW�.5E$ - 0�� � (W-v�1K]P,$�=?�!Ï#57Ò*$�IU$��6CE�!=?�!ÏCE�657$¾57Ò*�65�Ña=UI?IaÓ�0.< $¾57Ò!$�Ñ4$�=?Ï Òl5�5E0å57Ò!$�Ó�=U�*=UÓ�(!Ó�- � � ,7.c-�=?�ãÎ!CE$;Ì�=?K7$�I?м0 �*$�KM57$;Î�»K7$�$@îv=?Ï (*C7$ & �»=8�eP �nA!áOÖ 0 =?KaK7Ó��6I?IU$;C�5EÒ*�6� 0�� � (�57Ò!$;�º5EÒ!$�Ke5E$�Î*K7=?ë�$:Ña=UI?IcP9$�K7Ó��6I?IU$;C4� �*GÌr0 �><1$�CEÏ $��9Ìr$aÑa=?IUI,5E� Í $�Ó�(!IU57=?Î!IU$D57=?Ó�$;KM57$;Î*K�A>áOÖ 0 =JK)P9$�5eÑ)$;$�� 0 � � ( �6�*G E 0 � � ( 57Ò!$;��57Ò*$Ñ)$;=UÏ1Ò15RÑa=?IUI�01KEÌr=?I?I?�657$D� C701(!�*G � � ,/. P!(�5aÑa=?IUI�$;< $��l5E(*�6I?IUÐ�Ì�0 �>< $;C7Ï1$ �»î�=?Ï (!CE$ & �Ý=U�eÌ �nA�áOÖ0 =JK�Ó�0 CE$@57Ò9�6�#5eÑa=JÌr$@5EÒ!$�K7=Uë;$]06Ö 0 � � ( �Àî�=UÏ1(!C7$�& �Ý=U�eG �457Ò!$;�#57Ò*$ÙKe5E$�Î*K7=?ë�$]=JKK70�I?� C7Ï1$57Ò*�65a57Ò!$�Ñ4$�=?Ï Òl5a$��*G*KR(!Î3Ö»�6C757Ò!$;CRÖÝC701Ó � � ,7. 57Ò*� �3P,$rÖÝ01C7$1A!Û=?< $;C7Ï1$��*Ì�$@C7$�KM(!IU5EK;A

E ( ω )

ω

η > η opt

ω min

E ( ω )

ω

η < η opt

ω min a)

c)

E ( ω )

ω

η = η opt

ω min

E ( ω )

ω

η > 2 η opt

ω min

b)

d)

E ( ω )

ω

η = η opt

ω

∆ω

dE/d ω

ω c

ω min

ω c

ω min

d E( ω c ) d ω

� i � � iki �L<M7N ¹��*¹ � bM\�� i¦_Ej6s)� _WghfW_Wj.s�°Jd�bR� i�¿!_Ebe_Wj.s��k_n\�bej1i¦j1^@bM\�se_WgW©

äxÒ*�.5�=JK�57Ò!$#01Î�57=?Ó��6Ia<.�6I?(!$�06ÖD57Ò!$#I?$;� C7�!=?�!ÏåCE�657$ 0�� � (��Á"%$r5�(9K�â*CEKM5�Ìr01�*KM=JG�$;C57Ò!$ÙÌ;� K7$@=U� �rLhG�=?Ó�$��*K7=U01�%A �DK7K7(!Ó�=U�*Ï�57Ò*�65:� Ì;�6��P,$Ù�6Î!Î*C70'æ�=?Ó��657$�G�P>о��Ül(*�1G�CE�657=JÌÖÝ(!�*Ìn5E=U01�%- 0 � � ( Ì;�6�#P,$�G�$;C7=?< $�G�PlÐ3â*CWKM5D$ræ�Î*� �*G�=?�!Ï � =?� ��Þ��'Ð>IU01CK7$�CE=U$�K�6P,0 (!557Ò*$Ìr(!CEC7$;�l5RÑ4$�=?Ï Òl5�- ��DC

� � ���<� �&� � � ��7 ��� F � � � � �&� ��-�� � 7 �E � � F � � � F � F6�&����-�

� � F 7 565D54 � E ���ÑaÒ!$�CE$]Ñ4$](*K7$]57Ò*$�K7Ò!0 C757Ò*� �*G�� �� �����

� � � � �� ��� � K � � A,áOÖ � =JKÜl(*�1G�CE�657=JÌ@57Ò*$�K7$;Ìr01�*G

0 CWG�$�C]G�$;C7=?<.�.5E=U<1$�=JK]Ì�0 �*KM5E� �15]� �*G 57Ò!$�Ò!=?Ï Ò*$�C]01CEG�$;C:57$�CEÓ�K�<.�6�!=JK7Ò%A/ÛD=WV�$�CE$��l57=J�.5E=U�*ÏP90 57Ò#KM=JG�$�KRÑa=¦5EÒ3CE$;K7Î,$;Ìn5a5E0&� 5EÒ!$���Ï =?< $�K

� �&� ���� � � � �&� ��-�

� � 7 � � F ��-� � F6�&����-�� � F 5 � EXE �

Page 18: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

?>$r5757=?�!ÏB� � � � ,7.#�6�*G#�*0657=?�!Ï�5EÒ*�.5 � �&��� � ,7. � � � � � � -9Ñ)$��6CE$]IU$�Ö×5:�6Ö×57$�CDC7$��6C7LCE� �!Ï =?�!Ï�Ña=¦5EÒ

� � ,7.&� � � F � FD� � ��-�� � F � � + � � � ��-�

� � 5 � E � �&)0 Ó�Î*�6CE=?�!Ïå57Ò!=JK�Ña=U57Òì57Ò*$Á(!Î�G!�.5E$Á$;Ül(*�657=?0 � � E � �n-�Ñ4$�â*�*GT5EÒ*�.5¾Ñ)$ÁÌ;�6�TC7$�� ÌWÒì�Ó�=?�!=?ÓÙ(!Ó =U��01�!$�Ke5E$�Î�=¦Ö

0�� � (<� � F �&� � � �� � F � � + 5 � E � �

�®$�CEÒ*�6Î*K®� ��$;� K7=?$�C�ÑR�'Ð]570�01P�5E� =U��5EÒ!=?K)K7� Ó�$aC7$�KM(!IU5�=JK�=?IUI?(*KM57CW�.57$�G�=U��î�=?Ï (!CE$ & �»=U=U�nAÞRÒ!$¼P90 5M5E0 Ó Ï1CE� Î!Ò Î*IU0 5EK357Ò*$¼Ï CW� G�=?$��l5#06Ö�� �1K#�ãÖÝ(!�*Ìn5E=U01� 06Ö&� A ?>=?�*Ìr$ � =?KÜ1(9� G�CW�.5E=?Ì -v5EÒ!$#Ï CW� G!=U$;�15�=JK�KM=?Ó�Î!IUÐã� KM57CW�6=?Ï Òl5�IU=?�!$#Ña=U57ÒT<'� IU(*$3ë;$�CE0å�65�57Ò!$#Ó�=?�!=¦LÓÙ(!Óõ�6�*G ���� � � �

����.5�5EÒ!$Ì�(!C7CE$��l5�Ñ4$�=?Ï Òl5 � � A � FD� � � F6� =JK�K7=UÓ�Î!I?Ð]5EÒ!$K7IU01Î9$R06Ö�5EÒ!=?K

IU=?�!$]�6�9Gº=JK�Ìr01Ó�Î!(�57$�G3(*KM=?�!Ï�57Ò*$]Ke5W�6�*G!� CEG¾KMI?0 Î,$@ÖÝ0 CEÓÙ(*I?�

� F � � � F � � � �&� � � � � � � F ��� F � � ,/. 5 � E " �?>0 I?<>=U�!Ï�ÖÝ01C:� � ,7.º5EÒ!$���Ï1=U<1$;K4$;Ül(*�657=?0 � � E � �nA

äxÒ!=UI?$#5EÒ!$ÁI?$;�6CE�!=?�!ϼCW�.57$�5EÒ*�.5¾Ï =?< $;K�Ö»� KM57$;KM5¾Ì�0 �>< $;C7Ï1$��*Ì�$¾=JK 0 � � ( -�57Ò*$ÁI?� C7Ï1$;KM5IU$��6CE�!=U�*ÏTCW�.5E$å57Ò*�65 Ì��6� P,$¼(*K7$;G Ña=¦5EÒ!0 (!5 Ì�� (*KM=?�!ÏìG�=?< $�CEÏ $;�*Ìr$ú=JK �»� I?K70xKM$;$¼îv=?Ï6L(!C7$'& �Ý=U�eG �

0 � + � � E 0�� � ( 5 � E & �áOÖ1� =?K@�!0 5�$ræ!� Ìr57I?ÐÁÜ1(9� G�CW�.5E=?ÌÙ57Ò!$;� 5EÒ!$�Ò!=?Ï Ò!$;C@0 CWG�$�CD57$;C7Ó�K@=?� $�Ü1(9�.57=?0 � � E � �:�6CE$�!065�Î!CE$;Ì�=?K7$�I?мë�$;C70ú�6�*G � E � �Ù=?K�01�!I?Ðz�6�ì�6Î!Î!CE0'æ�=UÓ��657=?0 �%A�áh�TK7(*ÌWÒì� Ì;� K7$ -®=U5�Ó��'Ð5E�6Í1$]ÓÙ(*I¦5E=UÎ!I?$Ù=U57$;CE�657=?0 �*K�570¾IU0�Ì��657$�5EÒ!$�Ó�=?�!=?ÓÙ(!Ó $;< $��#ÑaÒ*$��Á(*K7=?�!Ï 0 � � ( -9Ò!0.Ñ4$�<1$�C�-Ìr0 �><1$�CEÏ $��9Ìr$DÌ;�6��Ke5E=UI?I%P,$]Ül(!=¦5E$@Ö»� KM5;A

áh� ÓÙ(*I¦5E=UÎ!I?$�G�=UÓ�$;�*KM=?0 �9K�-cG�$�57$�CEÓ�=U�*=U�!Ï 0�� � (a=?K@�¾P!=¦5@Ó�0 CE$�G�=UÕ�Ìr(!IU5�P9$�Ì�� (*KM$�57Ò*$C7=?Ï Òl5K7=?G�$�0 Ö� E � �4=?K���Ó��657CE=¦æ�� � +DÑaÒ!$;C7$�� =JKÌ;�6I?IU$�Gº5EÒ!$]Ø�$�K7K7=J�6��ÑaÒ!01K7$]Ìr01Ó�Î90 L�!$��l5EK�6CE$DÏ1=U<1$��3PlÐ

��,! � � FD�� � , � � � E (X�

Ña=¦5EÒ ��� � ���� �¼-*� �*G � $�Ü1(9�6I�570�5EÒ!$@570 5E�6I/�l(*ÓÙP,$�Ca06Ö®Ñ4$�=?Ï Òl5EK;A��=JK��Ó�$;�1KM(*C7$406Ö957Ò!$aÌr(*C7<.�.5E(!CE$)0 Ö ��A.áh��5eÑ)0@G�=?Ó�$��*K7=U01�*K;-'57Ò!$RI?=U�!$�K®06Ö�Ìr0 �9Ke5W�6�l5��ÖÝ01C®�:Ül(*� G!CE�657=JÌ�Ìr0lKe5��6CE$40.<'� I>=U��KMÒ*� Î9$a�1K�K7Ò!0.Ña�]=?��îv=?Ï (*C7$ (>A.ÞRÒ!$R$�=?Ï $;�>< $;Ìr5701CEK/06Ö

� Î901=U�l5)=?��5EÒ!$DG�=?CE$;Ìn5E=U01�*K�0 Ö�57Ò!$DÓ��Q�e0 C�� �*G�Ó�=?�!0 C4�.æ�$;K;A ÞRÒ!$D$�=?Ï $��><.�6I?(!$�K�Ó�$;� K7(!CE$57Ò!$]KM57$;$�Î!�!$�K7KR0 Ö�� �6I?0 �*Ï�5EÒ!$]Ìr01C7CE$;K7Î,0 �*G�=?�!Ï�$�=?Ï $��9G�=UCE$;Ìr57=?0 �%A������ ��.�%&� áh�z57Ò!$¾IU$�� KM5ÙÓ�$;�6�ãK7Ül(*� C7$��À"/Q�? �]�6I?Ï 01C7=U57Ò*Ó3-vÑ)$ºÒ*�'< $º� KM=?�!Ï I?$ºIJ�'Ð $;CIU=?�!$;� Ca�!$r5eÑ40 CEÍ�Ña=U57Ò�$�CEC701C4ÖÝ(!�*Ìn5E=U01�

�&����� � �E I9

J�6Kv+� � � F J , � , � � , � F � E * �

Page 19: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Eigenvectors of H

ω min,1

ω min,2

ν 1

ν 2

ω 1

ω 2

P

E

� \ � � {��L<M7N ¹��!¹ y9ikjl_Wg�d�°cfWd�j1gOsM\;j.s���©ÑaÒ!$�CE$!Ií=?K:5EÒ!$��>(!ÓÙP,$�C@0 Ö�57CW�6=?�!=?�!Ï�< $�Ìn5701CEK;A�ÞRÒ!$�Ø�$;KEK7=?� � =?� 5EÒ!=JK�Ì�� K7$�57(!CE�*K@0 (!557Ò!$�P,$@57Ò!$]KE�6Ó�$�� K45EÒ!$]Ìr0.<.� C7=J�6�*Ì�$Ó��.5EC7=Uæº0 Öv5EÒ!$�=U�*Î!(�5EK;-

� � �I J �� � � � + 5 � E - �

ÞRÒl(9K�-!$�� ÌWÒ�$�=?Ï $;�><'� IU(*$:0 Ö � =JK��6IJK70���Ó�$�� K7(!C7$�0 Öv5EÒ!$ÙÌ�0.<'� C7=J�6�9Ìr$:0 C�K7Î!C7$�� G¾06Ö®57Ò*$=U�!Î*(�5EK�� IU01�!Ï�57Ò!$]Ì�0 CEC7$�KMÎ,0 �9G�=U�*ÏÙ$�=?Ï $;�*G�=?C7$�Ìn5E=U01�3�1KaKMÒ!0.Ña�3=?�3î�=?Ï (!CE$'*!A

x 1

x 2

L<M7N ¹��*¹ �ld�b@seol_�y � �Á\;�k^�d�bei�seolZ�}cseol_�_Wik^�_Wj6�'_EfEsed;beg]\�j>�#_Wik^�_Wj6�;\;�k�l_Eg�d;°�� Z�_n\;gh� be_Ùseo1_ghq be_n\���d;°�seo1_ai¦j1ql� seg�i¦j�ikjlql� s�ghq>\�fW_�©

�K7=U�!Ï���KEÌ��6IJ�6CRI?$;� C7�!=?�!Ï�CW�.57$@=JKRÎ!CE0 P!I?$�Ó��.5E=?Ì:=?��ÓÙ(!IU57=?Î!I?$]G�=UÓ�$;�*KM=?0 �9K�A!äå$�ÑR�6�l50 5E0ÙP,$DIJ�6CEÏ $�K70]5EÒ*�.5RÌr01�l<1$�CEÏ $;�*Ìr$a=JK�Ö»�1Ke5R�6I?0 �!Ï@5EÒ!$@KMÒ*� IUI?0.ÑxG�=?C7$�Ìn57=?0 �9K�06Ö0� �»K7Ó�� IUI$�=?Ï $��><.�6I?(!$�KD0 Ö ���r-cÒ!0.Ñ)$;< $;C;-,=¦Ö 0 =JKD570>0�I?� C7Ï1$]57Ò*$�Ñ)$;=UÏ1Ò15WKÑa=?I?I�G�=?< $�CEÏ $��6I?0 �!Ͼ57Ò*$Ke5E$�$�ÎìG�=?C7$�Ìn5E=U01�*K'�ÝIJ�6CEÏ $3$�=?Ï $��><.�6I?(!$�KÙ0 Ö ���rA�Þ/0¼K7$�$¾57Ò!=JK�Ó�0 CE$3K7Î9$�Ìr=Uâ9Ì�� IUI?Ð -�I?$r5�(*K�6Ï1� =U�¾$ræ�Î*� �*G'��-!P!(�5a5EÒ!=JKR57=?Ó�$]�6P,0 (�5���Ó�=U�!=?ÓÙ(*Ó

�&� ���� �&��� � ,7. ��7 �E � � F � � ,7. � + � �� � � � � � F � � ,7. �-5 �#� � �

Page 20: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Û=/V,$;C7$;�l57=J�.57=?�!Ï��#� � �4� �*Gº(9KM=?�!Ï�57Ò*$�C7$�KM(!IU5a=U�35EÒ!$�(!Î�G!�.5E$�$;Ül(*�.5E=U01� � E � ��Ï1=U<1$;K���/.�7 � �<� ���/.%� F10 � � �/.%�

� � �#�"���� ���/.%� F10 � � � � � � ����.%� F � � ,/. �-5 �#� E �

?>(!P�5ECE�1Ìn57=?�!Ï&� � ,7.¾ÖÝC701Ó�P,0657Ò#K7=?G!$;KRÏ =?< $�K� ����.�7 � � F � � ,/. �<� ��� F 0 � ��� � ,7. �%�6� ����.%� F � � ,/. �-5 �#� � �

áOÖ/57Ò*$:Î*C7$�Ö»� Ìn5E0 CG��� F 0 � ��� � ,7. �%�)=JKa�ÙÓ��.5EC7=Uæ�57CW�6�9KeÖÝ01C7Ó��.5E=U01��57Ò9�.5a�6I?Ñ4�'Ð�K4KMÒ!CE=?�!Í�K�Á< $�Ìn5E0 C �»=ÀA $ A®=U5EK�$�=?Ï $;�><'� IU(*$;KÙ� IUIRÒ*�'<1$�Ó��6Ï1�!=¦5E(*G�$¾IU$�K7K�57Ò9�6� ����5EÒ!$��ç57Ò!$3(!Î�G!�.5E$$;Ül(*�.5E=U01�¾Ña=?I?I/Ìr01�l<1$�CEÏ $1A

Ø�0.Ñ G�0>$;KÙ57Ò!=JK�Ò!$�I?ÎzÑa=U57ÒãÌWÒ*0l0lKM=?�!Ï#5EÒ!$ºI?$;� C7�*=U�!Ï CE�657$;K �ºáeG�$;� IUI?ÐúÑ)$�ÑR�6�l5�G�=¦Ö×LÖÝ$�CE$��l5ºI?$;�6CE�!=?�!ÏúCE�657$�K��6I?0 �!Ïú5EÒ!$ G�=/V�$�CE$��l5º$�=?Ï $;�*G�=?C7$�Ìn5E=U01�*K�A�ÞRÒ*=?Kº=?K�K7=?Ó�Î*IU$Á=UÖ�57Ò*$$�=?Ï $��9G�=UCE$;Ìr57=?0 �*K:�6CE$ÙI?=?�!$;G (!Î Ña=U57Ò 57Ò!$�Ì�0l01CEG!=U�*�657$��.æ�$;KD06Ö�5EÒ!$�Ñ)$;=UÏ1Ò15WK�A,áh�úKM(*ÌWÒ �Ì�� K7$ ->5EÒ!$�Ñ4$�=?Ï Òl5EKR�6CE$@(!�*Ì�0 (!Î!I?$;G3�6�9G¾Ñ4$�Ì�� �3�1K7K7=UÏ1�º$�� ÌWÒ3Ñ)$;=UÏ1Ò15a=U5EKR0.Ña�3IU$��6CE�!=?�!ÏCE�657$P9� K7$;G�01��57Ò*$:Ì�0 CEC7$�KMÎ,0 �*G!=U�!ÏÙ$�=?Ï $��><.�6I?(!$1AlØ0.Ñ)$;< $�C�- =UÖ%5EÒ!$DÑ4$�=?Ï Òl5WK)� C7$:Ìr0 (*Î!IU$�G57Ò!$;��Ñ)$aÓ�(*Ke5�â*CEKM5�CE065E�657$ � K7(*ÌWÒ�57Ò*�65 � =?K�G�=?� Ï 01�*�6I[-.=ÀA $ A 57Ò!$DÌr0>0 CWG�=?�*�.5E$R�6æ�$;K�I?=U�*$(!Î�Ña=U57Ò�5EÒ!$$;=UÏ1$��*G!=UCE$;Ìr57=?0 �*K �»K7$�$Dî�=?Ï (!CE$ (.P �nA>ÞRÒ!=JK�=JK�57Ò!$DÎ!(!CEÎ90lKM$D06Ö/G�=J�6Ï10 �*� IU=?ë�=?�!Ï57Ò!$]Ø$;KEKM=J�6��G�=JK7Ì�(*K7K7$;G¾$;� C7I?=?$�C�A

"%$r5���P,$@57Ò!$�CE065W�.57=?0 �¾Ó��.57CE=Uæ¾KM(9ÌWÒº5EÒ*�.5� ��� ��� + �#� � �

ÑaÒ!$�CE$ � =?K�G!=?� Ï 0 �9�6Ic�6�*G�� + � ���9A!ÞRÒ!$]Ìr0lKe5RÖÝ(!�*Ìr57=?0 �357Ò*$��ÁÌ�� �¾P,$�ÑaCE=¦5757$��#�1K

�&����� � �&��� � ,7. ��7 �E� � � F � � ,7. � + � +� � � � �� � � � � +� � ����� F � � ,7. � � 5�#� " �

QÁ�6Í>=?�!Ïã�ãÌWÒ*�6�!Ï1$Á06Ö�Ìr0>0 CWG�=?�*�.5E$;K�5E0 � � ����� F � � ,7. �¾KM=?Ó�Î!IU=Uâ*$�K�5EÒ!$å� P90.<1$$;Ül(*�.5E=U01�º5E0

�&� � �� �&� � ��7 �E � + � � �#� & ��6�*Gº57Ò!$@5ECE� �*KMÖÝ0 CEÓ�$�Gº(*Î,G!�657$�$�Ü1(9�.57=?0 �3P,$;Ìr01Ó�$;K

� �/.�7 � �<� ��� F 0 � � � ��.%�-5 �#� (X���065E$�57Ò9�.5�� F 0 � =?KDG�=J�6Ï 01�*�6IcÑa=U57Ò G�=?� Ï 01�*�6I/Ì�0 Ó�Î,0 �!$;�15WK � F 0�� , A�ÞRÒ!=?K$;Ül(*�657=?0 �Ña=UI?I%Ìr01�l<1$�CEÏ $�=UÖ � � F 0�� , � � � ->=ÀA $ A 0 � F� � ÖÝ0 Ca�6I?I �WA>áOÖ�Ìr01�*KM57CW�6=?�!$;G�5E0�Ò*�'< $:� nÿ��lû ���K7Ì;�6IJ�6CRIU$��6CE�!=?�!Ï�CW�.5E$:ÖÝ0 C��6I?I%Ñ4$�=?Ï Òl5EK)5EÒ!$���Ñ4$�ÓÙ(*KM5aC7$�Ü1(*=UCE$

0 � E� � + � �#� * �

=U�ì0 CWG�$;C�5E0z�'< 01=?GãG�=U<1$�CEÏ $;�*Ìr$1-�ÑaÒ*$�CE$ � � + � =JK�57Ò!$#IJ�6CEÏ $�Ke5�$;=UÏ1$��><.�6I?(!$¾0 Ö �zA�î!01CÖ»� KM57$;KM5Ìr01�>< $�CEÏ $;�*Ìr$DÑ)$@Ò*�'<1$

0�� � (�� �� � + � 5 �#� - �

Page 21: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

áOÖ � � ,/.3=JK���IU0 5KMÓ�� IUI?$�CR5EÒ*�6� � � + ��57Ò!$;��Ì�0 �>< $;C7Ï1$��*Ì�$DÑa=?I?I%P9$�<1$�CEкK7IU0.Ñ �6I?0 �!Ï�57Ò*$� � ,7.ÙG�=?C7$�Ìn5E=U01�%A;áh�ÙÖ»� Ìr5;-.Ìr01�l<1$�CEÏ $;�*Ìr$�5E=UÓ�$4=?K�Î!CE0 Î,0 C757=?0 �9�6I65E0�5EÒ!$aÌr0 �9G�=¦5E=U01�]�>(!ÓÙP,$�C� � � � + � � � � ,/.ÁK70�5EÒ*�.5:=¦5:=?K:G�$;K7=?CE� P!IU$]570�Ò*�'<1$Ù� KDKMÓ�� IUI��6�Á$;=UÏ1$��><.�6I?(!$ÙK7Î!C7$�� G#�1KÎ90lK7K7=UP*IU$1A

Ø�0.Ñ4$�< $;C;-cK7=U�9Ìr$�Ñ4$�Ò*�'< $�CE065E�657$�G � 5E0#P9$º� IU=?Ï �*$;GåÑa=¦5EÒú57Ò!$¾Ìr0>0 CWG�=?�*�.5E$��6æ�$;K;-��� (X�aÌr01�*K7=?KM5EK��1Ìn5E(*�6I?IUÐ306Ö �à=U�*G!$�Î,$��*G�$;�l5 �rLhG�=?Ó�$��*K7=U01�*�6I/$;Ül(*�.5E=U01�*K;A*ÞRÒ!$�CE$rÖÝ01C7$1-!Ñ4$Ì��6�ãÌWÒ!0>01K7$º�ÁI?$;� C7�!=?�!ÏÁCW�.5E$�ÖÝ01C�$�� ÌWÒ¼Ñ4$�=?Ï Òl5�=?�*G�$;Î9$;�*G�$��l5�06Öa5EÒ!$¾0 57Ò!$;CEK;A�ä $3KM$;$57Ò*�65a57Ò!$�01Î�57=?Ó��6IcCW�.57$:ÖÝ01CR57Ò!$ � (�� Ñ4$�=?Ï Òl5 � , =JK 0 � � ( � , � +� � A���� ����� � .0%1

@B"03�%1�2' �A%(� � �&'�� îv=?Ï (*C7$ � � G�=JKMÎ!IJ�'Ð�K��åK7$r5�0 Ö � ��� $�æ!�6Ó�Î!I?$;K�G�CW�'Ña�zÖÝCE0 Ó 5eÑ408@�6(*KEKM=J�6��G�=?KM57CE=?P!(�57$�GÙÌ�I?�1K7K7$;KvÌr$��l5E$�CE$;G]�65�ÝL � A � - L � A * ��� �*G � � A � - � A * �nA'ÞRÒ!$)$�=?Ï $��><.�6I?(!$�K06Ö�57Ò*$)Ìr0.<.� C7=J�6�*Ì�$�Ó��.57CE=Uæ@� C7$ � A * � �6�*G � A � � &*Aräå$�57CW�6=?�Ù��K7=?�!Ï I?$�IJ�'Ð $�C/IU=?�!$;� C/�!$�5eÑ)01C7ÍÑa=¦5EÒ E =?�!Î!(�5WK�- ��0 (!57Î!(�5�- E Ñ4$�=?Ï Òl5WK�-c�6�9G���P!=J� K��»K7$�$�î�=?Ï (!CE$B�#- �;�(9KM=?�!Ï357Ò!$�"vQ�?�6I?Ï 0 CE=U57Ò!Ó =?�ìP*�.5WÌWÒãÓ�0�G�$1A)î�=UÏ1(!CE$ � �#G�=?K7Î!IJ�'Ð�K�57Ò!$#Ñ4$�=?Ï Òl5�57CW�P�e$�Ìn5701C7Ðz� �*GT$�CEC701CG�(!CE=U�!Ï�I?$;� C7�!=?�!Ï3ÑaÒ!$��ú(*K7=?�!Ï#�¾I?$;� C7�*=U�!Ï3CW�.57$�K:06Ö 0 � �X5 "¾� �*G E A "�A���0 57$�57Ò9�.5�57Ò*$IU$��6CE�!=U�*Ï�CW�.5E$��ÀKM$;$ A�Ü,A � * � 0 � + � � E � � � + � � E �$5 * � � E 5 � *�Ña=?IUI/Ì;�6(*K7$�G�=?< $�CEÏ $;�*Ìr$�K� KR=JKR$�<>=?G!$��l5aÖÝ0 C 0 � E 5 "!A

ω1ω0

ω2

χ0

y

χ1

L<M7N ¹��*¹ �6ikZ�ql�k_R�kikjl_n\rb4j1_Es[��d;be��©−1.4 −1.2 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4

−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

L M/N ¹�O��9¹ u��®d]fW�¦\�ghgh_Wga� bM\W��j�°Ubed�ZÚ^'\��lghghi¦\;j¾�1ikgOshbei�|{l�1seikd;jlg�fW_Wj.se_Ebe_W��\�s � |À� © £ } |À�6© ¤ �)\�j>� � �6© £1} � © ¤��7©

îv=?Ï (*C7$'� E K7Ò!0.Ñ�K:5EÒ!$ºKE�6Ó�$�$ræ!�6Ó�Î!I?$�(*K7=U�*ÏÁKe5E0>ÌWÒ9� KM57=JÌ�=?�*KM57$�� Gå06ÖaP*�65EÌWÒúÓ�0>G!$IU$��6CE�!=U�*Ï*A6Ø�$;C7$1- �@I?$;�6CE�!=?�!Ï@CE�657$R06Ö 0 � � 5 E =JK�(*K7$;GcAlBD�*$�Ì��6��K7$�$R5EÒ*�.5�5EÒ!$a57CW�P�e$�Ìn5E0 CEÐ=?K�ÓÙ(9ÌWÒç�!01=?K7=U$;CÙ5EÒ*�6�ã=U�ãP*�.5WÌWÒçÓ�0�G�$�KM=?�*Ìr$30 �*IUÐz� �ç$;KM57=?Ó��.5E$¾0 Ö�57Ò!$3Ï CW� G!=U$;�15�=?K(*KM$�Gã�65�$;� ÌWÒT=U57$�CW�.5E=U01�%A�ÞRÒ!$#Ìr0lKe5�=JK�Î!I?065M5E$;Gì� K�� ÖÝ(!�*Ìr57=?0 �ì06Ö:$�Î,0�ÌWÒ%A<���ì$�Î,0�ÌWÒÒ!$�CE$@=?K�K7=?Ó�Î*IUкG!$râ*�!$�G�� KG� ��� =?�!Î!(!5Î!CE$;K7$��l5E�657=?0 �*K4ÑaÒ!=JÌWÒ%->ÖÝ0 C�KM570�ÌWÒ*�1Ke5E=?Ì:I?$;� C7�!=?�!Ï9-

Page 22: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Weight space

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10−20

−15

−10

−5

0

epochs

Weight space

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10−20

−15

−10

−5

0

epochs

� \ � � { �

L<M7N ¹PO�O>¹ � _Wik^�o.s:shbM\��O_EfEsed;bh� \�j>�#_Ebhbed;b@fW�1be�'_��1� beikjl^��k_n\�bej1i¦j1^º°?d;b � \ ��� í~�� ¡º\;jl� � {��� z¢ � ¡6©

Page 23: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Ìr0 CECE$;K7Î901�*G!Kc5E0 � � � Ñ)$;=UÏ1Ò15�(!Î�G!�.5E$;K;A.áh�ÙP*�65EÌWÒ%-.�6��$�Î,0�ÌWÒ�Ìr01C7CE$;K7Î,0 �*G!K%570:0 �!$4Ñ4$�=?Ï Òl5(!Î,G*�.57$1A

Weight space

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10−20

−15

−10

−5

0

epochs

batch

L<M7N ¹PO �*¹ � _Wik^�o.sÚshbM\��O_WfEsed�bh� \;jl�é_Ebhbed;bfE�1be�'_Ù�1� beikjl^�gOsed.fMo>\�gOsei¦f��k_n\�bej1i¦j1^�°Jd;b � � � ¢6©

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10

−20

−15

−10

−5

0

Weight space

−2 −1.6 −1.2 −0.8 −0.4 0 0.4 0.8 1.2 1.6 2 2.4

−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

epochs

L<M7N ¹�O �*¹ � _Wik^�o.s3shbM\��O_Wf7sed;beik_WgÁ\;jl�x_Ebhbed;beg°Jd�b�~E|e~E|e~�jl_EsÀ�®d�be��shbM\;ikjl_n�Á�lghikj1^3gOsed.fMo>\;gO|seikfa�k_n\�bej1i¦j1^1©

� *C.0��"�.0� �&%(' �A%(� � �:'�� î�=UÏ1(!C7$�� � K7Ò!0.Ñ�K)57Ò*$]�6CWÌWÒ!=¦5E$;Ìr57(!CE$:ÖÝ0 C���< $;C7Ð�K7=?Ó�Î*IU$�Ó�(!I¦L57=?I?�'Ð1$�C®�!$r5eÑ40 CEÍ�A.áO5�Ò*� K�a=?�!Î!(!5;-"�aÒ!=JG!G�$;�%- � �*GB�401(�57Î*(�5��!0�G�$1A6ÞRÒ!$�CE$��6CE$ E Ñ4$�=?Ï Òl5WK�6�*G E P!=J� K7$;K;A*ÞRÒ!$�� Ìn5E=U<.�657=?0 �3ÖÝ(!�*Ìr57=?0 �#=JK ��� � �1� � 5 ($�c5E� �!Ò �%� E ��� � � �nA9ÞRÒ*$�57CW�6=?�!=?�!ÏKM$�5:Ìr01�l5E�6=?�*K � � $�æ�� Ó�Î!IU$�KaÖÝCE0 Óï$;�1ÌWÒ#06Ö E Ì�I?�1K7K7$;K;A924065EÒÁÌrIJ� KEKM$�K� C7$Ù8@� (*K7K7=J�6�ÁG�=JKeL57CE=UP!(!57$;GºÑa=¦5EÒ3KM5E� �*G!� CEG�G!$�<>=?�657=?0 � � A � A�&)IJ� KEK �DÒ*�1K)�ÙÓ�$;�6�¾06Ö%L �D� �*GºÌ�I?�1K7K E Ò9� K4�Ó�$��6�¾06Ö 7 �1A>Þ��6CEÏ $�5)<.�6I?(!$�K4� C7$L#�DÖÝ0 CRÌrIJ� KEK��:� �*G 7 �ÖÝ0 CaÌrIJ� KEK E A�î�=UÏ1(!C7$!� ��KMÒ*0.Ñ�K57Ò!$]KM570�ÌWÒ*�1Ke5E=?ÌD57CW�P�e$�Ìn5E0 CEÐ�ÖÝ0 CR57Ò*$�$ræ!�6Ó�Î!I?$ A

Page 24: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

ω1

ω0

ω2

ω3

χ

y

L<M7N ¹PO��,¹ u®ol_aZ�ikjlikZ]\��*ZD�l��seik�¦\n�'_Eb�jl_EsÀ�®d�be�!©

���� � 3 � *�� � ')�!3� �> �&'� �2��"0�<3C �!3 9 �?'-')�:' ��*C'->��!� % � ')�!3C �>��:'� �!��"0�<3� �% #�"� �"0��% 9

ä $#Ì�� �ç(*K7$¾57Ò!$�CE$;K7(!I¦5WK�06Ö�5EÒ!$�Î!C7$;<>=U01(*K�K7$;Ìr57=?0 �ç570B�e(*KM57=UÖÝÐçKM$;< $;CE� I)0 Ö�57Ò!$35EC7=JÌWÍ�KG�=?KEÌr(9K7K7$;G¾$;� C7I?=U$;C;A

��* � � ')�!� �=����%� %1�!3� >�')� ����% "�3�� *�� #!�2')"�� � .�%1

ÞRÒ!$DCE$;�1KM01��ÖÝ0 CR5EÒ!$��6P,0.< $57CE=?ÌWÍ�=JK457Ò*�65��Ù�!01�!ë�$;C70�Ó�$;�6�3=?�º5EÒ!$@=U�!Î*(�5a<'� C7=J�6P*IU$�KÌrCE$;�.5E$;K��$5���ü�� �Uý.ü[û �$�=?Ï $;�l<.� IU(!$1A*ÞRÒ!=JK�Ó�$;� �*KR5EÒ!$ÙÌr01�*G�=U57=?0 �#�>(!Ó�P9$;C�Ña=UI?I/P,$]IJ�6CEÏ $1-=ÀA $ A.5EÒ!$Ì�01KM5�K7(!C7Ö»� Ì�$aÑa=UI?I!P,$�Ke5E$�$�Î�=?��KM01Ó�$RG!=UCE$;Ìr57=?0 �*K�� �*G�KMÒ9�6I?IU0.Ñã=U��065EÒ!$�CWK�K70@57Ò*�65Ìr0 �><1$�CEÏ $��9Ìr$ÙÑa=?I?I�P9$�<1$�CEÐÁKMI?0.Ñ�AcÞRÒ!$�KM01IU(�5E=U01� =JK:570#K7=UÓ�Î!I?ÐÁÎ!CE$�Î*C70�Ìr$�K7KD5EÒ!$�=U�!Î*(�5EKPlоKM(!P!57CW� Ìn5E=U�*Ï�5EÒ!$�=?C�Ó�$;� �*K;A

î!0 C���K7=U�*Ï I?$:I?=?�!$;� Ca�!$�(!CE0 �/->57Ò!$�$;=UÏ1$��>< $�Ìn5701CEK40 Ö/57Ò*$]Ø�$;KEK7=?� � �»Ña=¦5EÒ�Ó�$��6�*KaK7(!P�L57CW� Ìn5E$;G ��Î901=U�l5�� IU01�!Ï�57Ò!$¾Î!CE=U�*Ì�=UÎ9�6I��6æ>$�K�06ÖR5EÒ!$¾Ì�IU01(*Gú06ÖR57CW�6=?�!=?�!Ï#< $�Ìn5E0 CWK&�ÝCE$;Ì;�6I?Iîv=?Ï (*C7$ * �nA>áh�!Î*(�5EK45EÒ*�.5RÒ*�'< $:�ÙIJ�6CEÏ $<'� C7=J�.5E=U01��=?�3K7Î!CE$;� Gº�6I?0 �!Ï�G�=WV�$�CE$��l5aG�=?C7$�Ìn57=?0 �9K06Ö%5EÒ!$D=?�!Î!(!5RK7Î*� Ì�$Ña=?IUI�Ò*�'<1$�]IJ�6CEÏ $DÌr0 �9G�=¦5E=U01�º�>(!Ó�P9$;C4� �*G�K7IU0.ÑxI?$;� C7�!=?�!Ï9A ���9G¾K70Ñ)$@CE$;Ìr01Ó�Ó�$;�*G C

�A�&' ��!.0"�� % ����%$#!�2')"0�!3�� %1 =�<>����C%A"�3�� *�� #!�2')"�� � .�%1 �

áOÖ:5EÒ!$Á=?�!Î!(�53<.�6CE=?� P!IU$�K��6CE$ÁÌr01C7CE$�IJ�.5E$;Gc-®5EÒ!=?KºÑa=UI?I�*065ºÓ��6Í1$�57Ò!$ $�CEC701C�KM(!C7Ö»� Ì�$KMÎ!Ò*$�CE=?Ì;�6I[-�P!(�5�=U5�Ña=UI?I%Î,01KEKM=?P!I?Ð�C7$�G�(*Ì�$:=U5EK�$;Ì�Ì�$��l57CE=JÌr=U5eÐ A

&)0 CEC7$;I?�657$;G�=U�!Î*(�5a<'� C7=J�6P*IU$�K)(9KM(*� IUI?Ð�Ì�� (*K7$D5EÒ!$@$�=?Ï $;�l<1$;Ìr570 CWK�0 Ö � 570�P,$:CE065W�.5E$;G�'Ñ4�'Ð:ÖÝCE0 Ó 57Ò*$�Ìr0>0 CWG�=?�*�.5E$)�.æ�$�K1�Àî�=UÏ1(!C7$ (6�<1$�CWKM(*K ('P �%5EÒ>(*K®Ñ)$;=UÏ1Òl5�(!Î�G!�657$;K�� C7$4�!0 5G�$;Ì�0 (!Î!I?$;G%A*Û$�Ìr01(!Î!I?$;G3Ñ)$;=UÏ1Òl5EKRÓ��6Í1$D5EÒ!$Áè70 �!$@I?$;�6CE�!=?�!Ï�CE�657$@Î,$�C�Ñ4$�=?Ï Òl5Eê�Ó�$r5EÒ!0�G0 Î�5E=UÓ��6I[-�5EÒl(9K�-�Ñ4$@Ò*�'< $:57Ò*$@ÖÝ0 I?IU0.Ña=?�!Ï�57CE=?ÌWÍSC

� %(� �:'-')%1.0�2��%A����% "�3�� *�� #!�2')"�� � .�%1 �

��0.ÑõKM(!Î*Î90lKM$�5EÒ*�.5�57Ò!$º=U�!Î*(�5�<.�6CE=J�6P!I?$;K�0 Ö��#�!$�(*C701�¼Ò*�'< $�P,$�$;�çG�$;Ì�0 CEC7$;I?�657$�Gc-57Ò!$]Ø$;KEKM=J�6�3ÖÝ0 CR5EÒ!=JK��!$;(!C701�¾=JKR5EÒ!$��#G�=J�6Ï10 �*� I%�6�*G3=U5EKa$;=UÏ1$��><.�6I?(!$;KRÎ,0 =?�l5�6I?0 �!Ï�57Ò*$Ìr0>0 CWG�=?�*�.5E$º�.æ�$;K;A/áh�TKM(9ÌWÒç� Ì;� K7$�57Ò!$3Ï CW� G!=U$;�15�=?K��!065�57Ò!$3P,$;KM5�G�$�K7Ì�$��l5�G�=?CE$;Ìn5E=U01�� K�Ì��6�¼P,$�KM$;$��z=?�çî�=UÏ ('P%A��45Ù5EÒ!$3Î901=U�l5 �®-®�6�ã�6CEC70.ÑõKMÒ*0.Ñ�K�57Ò9�.5�Ï1CE�1G�=?$��l5�G�0>$;K�!065@Î901=U�l5@570.ÑR�6CWG!K�57Ò!$�Ó�=U�*=UÓ�(!Ó�A%Ø�0.Ñ4$�< $;C;-,=UÖ�Ñ)$�=?�*Ke5E$;�1G �1K7K7=UÏ1�Á$;�1ÌWÒÁÑ4$�=?Ï Òl5:=U5EK0.Ña� I?$;�6CE�!=?�!Ï3CE�657$'�Ý$;Ül(*� I�57Ò!$�=?�l<1$�CWKM$�06Ö�57Ò!$�Ìr01C7CE$;K7Î,0 �*G�=?�!Ͼ$�=?Ï $��><.�6I?(!$��a5EÒ!$��å57Ò*$

Page 25: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

G�$;KEÌr$;�15�G�=?CE$;Ìn5E=U01�¾Ña=?I?IcP9$�=?�º5EÒ!$]G�=?C7$�Ìn5E=U01�º06Ö�57Ò*$:0 57Ò!$;C��6CEC70.Ñì5EÒ*�.5aÎ,0 =?�l5EKaG�=?C7$�Ìn5EIUÐ570.ÑR�6CWG!K)57Ò!$�Ó�=?�!=UÓ�(!Ó'C

# �% � �% ���!'-�!��% .�%1�2')3C"�3C4 ')�2��% > �&' %1�!�-��� %1"�4&�;� �

� �åøMó �(�>Ë7Ê�ó®ø �lò/Ê�Ç�ÄDÈ Ç�Æ*È�ò/ÆTÇ ��Å!Ë��õËeùcó�Å!ËeÇ)Ä�� òcÅ���ÇaÈ?�

áh�Ù5EÒ!$4ÖÝ0 I?IU0.Ña=?�!Ï:Ñ)$RÑa=?IUI�P!CE=?$��*Ð�=?�l57CE0>G!(*Ìr$457Ò*$ �$�ÑR5E0 �%-6Ì�0 �P�e(!Ïl�.5E$)Ï1CE�1G�=U$;�l5;-68@�6(*KEKML��$�ÑR5E0 �%-l"%$;< $��>P,$�CEÏ�QÁ�6CWÜl(*�6CWG>5��6�9GÙ5EÒ!$ �D(*�1KM=UL ��$�ÑR5E0 �@�»2Rî�8 ?"�®Ó�$r5EÒ!0�G@�»K7$�$� I?K70� � � - � � - �*- " �*�nA����� �A% � ���<3 8 .04<�&'-"0����

Þ/0�Ï $�5@� � (*�*G�$�CWKM5E�6�9G�=U�*Ͼ0 Ö�57Ò*$ �$�ÑR5E0 �åÓ�$�57Ò!0�GåI?$r5�(*K@CE$;Ì�� Î!=U57(!IJ�.5E$Ù5EÒ!$�C7$�KM(*I¦5WKÖÝC701Ó�K7$;Ìn5E=U01� "�A7� AX�DK7K7(!Ó�=U�*Ï]��Ü1(9� G�CW�.5E=?Ì4I?01KEK�ÖÝ(!�*Ìn5E=U01�&� �»K7$�$ A)Ü9A8� E � �;�®� K�G�$;Î!=JÌn57$�G=U��î�=UÏ1(!C7$ & �Ý=?=8�r-6Ñ)$RÌ�� ��Ì�0 Ó�Î!(�5E$457Ò!$aÑ4$�=?Ï Òl5®(!Î,G*�.57$�� IU01�!ÏD57Ò!$RI?=U�!$�K�06ÖSA�Ü,A8� E ���OL-� E � �

��� � 0 � FD���� F �� + � �

��� � 0 ��� � � � + � ���� � ��� �

ÑaÒ!$�CE$ 0 Ó�(*KM5]570#P,$ºÌWÒ!0lKM$;�å=?�ú57Ò*$�CE� �!Ï $ � � 0 � �ºKM=?�*Ì�$&� =?K�=?�¼Î!CW� Ìr57=JÌr$��!0 5Î9$;CMÖÝ$�Ìn57I?Ð#Ül(*� G�CW�.5E=?Ì A!áh�#5EÒ!=?K$;Ül(*�657=?0 �#=?��ÖÝ0 CEÓ��.57=?0 �Á� P901(�5�5EÒ!$�Ø$;KEKM=J�6� �é=JK�5W�6Í1$��=U�l570��1Ì�Ìr01(!�l5;A>áOÖ/5EÒ!$�$�CECE0 C)ÖÝ(!�*Ìn5E=U01�¾ÑR� KRÜl(*�1G�CE�657=JÌ01�!$�Ke5E$�Î3Ñ)01(!IJGºP,$�KM(�Õ�Ì�=U$;�15R5E0Ìr0 �><1$�CEÏ $ A

�K7(*�6I?IUÐç57Ò!$ $��!$;C7Ï1ÐãK7(!CMÖ»�1Ìr$Á� C701(!�*Gç57Ò!$ Ó�=?�!=?ÓÙ(!Ó =JK�CW�.57Ò*$�C�$�I?I?=UÎ*K70 =JGc-)0 C�=?�57Ò!$�$ræ>5EC7$;Ó�$�I?=UÍ1$��º5E�1Ìr0�KMÒ!$;IUI[-cG�$�Î,$��9G�=U�*Ï301� 5EÒ!$�Ìr0 �9G�=¦5E=U01�!=?�!Ͼ0 Ö�5EÒ!$�Ø�$;KEK7=?� �%A �ÑaÒ!=¦5E$��!=?�!Ïa5ECE� �*KMÖÝ0 CEÓ3-nÑ4$�I?I Í>�!0.Ña�:ÖÝCE0 Ó KM=?Ï �*� I6Î!CE0�Ìr$;KEK7=U�!ÏRI?=¦5E$�CW�.5E(!C7$ � E - �1Ì�� ��ÌWÒ*�6�!Ï1$57Ò!=JK�$;IUI?=UÎ9KM01=?G�K7Ò*�6Î,$R570Ù��K7Î!Ò!$;C7=JÌ�� I!KMÒ9�6Î,$R57Ò!CE0 (!Ï1Ò��B� � � +��EF � �»K7$�$î�=?Ï (!CE$ ��":� �*GA�Ü,A7�#� � �%�nAP?>0D57Ò!$4=?�l<1$�CWKM$)Ø�$;KEK7=?� �Ù=?� A�Ü,A8� � � ��P*�1KM=JÌ�� IUI?Ð�KMÎ*Ò!$�CE$;Kv0 (�5�57Ò*$)$;C7CE0 C�KM(!C7Ö»� Ì�$IU0�Ì�� IUI?Ð AlÞRÒ!$aÖÝ01IUI?0.Ña=U�*Ï@5eÑ)0Ù� Î!Î!CE01�1ÌWÒ!$;K�Ì;�6��P9$:KMÒ*0.Ña��5E0ÙP,$�$;Ül(!=?<.�6I?$��l5DC �»� ��(*K7$a57Ò*$��$�ÑR5E0 �Á� IUÏ10 CE=¦5EÒ!Óí=?�Á�6�#(!�l5ECE� �*KeÖÝ01C7Ó�$�G¾Ñ4$�=?Ï Òl5K7Î*� Ì�$]�6�*G �»P ��G!0�(*K7(*�6I%Ï1CE�1G�=?$��l5G�$;KEÌr$;�15a=?�#��ÑaÒ*=¦5E$��!$�G#Ìr0>0 CWG�=?�*�.5E$:K7Ð�Ke5E$�Ó �ÀKM$;$]îv=?Ï (*C7$���" � � ��- �OA?>(!Ó�Ó��6CE=Uë;=U�!Ï9->57Ò!$ �$�ÑR5E0 �#�6I?Ï 01C7=U57Ò!Ó�Ì�0 �>< $;C7Ï1$;K)=U��01�!$]Ke5E$�Î�=UÖv5EÒ!$�$�CEC701C4ÖÝ(!�*ÌnL57=?0 �ç=JK�Ül(*� G!CE�657=J̺�6�9G �Ý(*�!IU=?Í $�Ï1CE�1G�=U$;�l5�G�$�K7Ì�$��l5 �Ù=¦5�=JK�=U�><.�6CE=?� �l5�Ña=U57ÒTC7$�KMÎ,$;Ìr5�5E0IU=?�!$;� C�57CW�6�*KMÖÝ0 CEÓ��.57=?0 �9KÙ0 Ö�57Ò!$�=?�!Î!(�5�<1$;Ìr570 CWK;A�ÞRÒ!=JK�Ó�$;�6�9K�57Ò*�65�5EÒ!$#Ìr0 �><1$�CEÏ $��9Ìr$57=?Ó�$#=JK��!0 5��QV,$�Ìn5E$;GTP>ÐçKMÒ*=¦Ö×5WK�-�KEÌ�� IU=?�!ϼ�6�*GãCE065E�657=?0 �ã06ÖD=?�!Î!(!5�< $;Ìr5701CEK;A�Ø�0.Ñ4$�< $;C0 �!$¾06Öa57Ò*$ºÓ��6=?�çG�CW�'ÑaP*� ÌWÍ�K�=JKÙ57Ò*�65��6� �� � Ø$;KEKM=J�6�zÓ��657CE=¦æúÓÙ(*KM5�P,$�Ke5E0 CE$;G�6�*G�=?�>< $�C757$�Gc-!ÑaÒ!=JÌWÒ�5E� Í $�K�����ÁV��aÎ,$�C�=U57$;CE�657=?0 �*K�6�*G�=JKa5EÒ!$�CE$rÖÝ01C7$�=?Ó�Î!CE�1Ìn5E=?Ì;�6IcÖÝ01CÓ�01C7$@5EÒ*�6�#��ÖÝ$;Ñ <.� C7=J�6P!I?$;K;A ?>=?�*Ìr$]57Ò!$]$;C7CE0 C4ÖÝ(!�9Ìn57=?0 �#=JK�=?�#Ï $;�!$�CW�6Ic�*0 ��LhÜl(*� G�CW�.5E=?Ì -57Ò!$;C7$�=JK��!0�Ï (9�6CW�6�l57$;$�0 Ö4Ì�0 �>< $;C7Ï1$��*Ì�$ A�áOÖ)5EÒ!$ºØ$;KEKM=J�6�å=?K]�!065]Î,01K7=¦5E=U<1$�G�$râ9�!=¦5E$��Ý=UÖ=¦5:Ò*� KDKM01Ó�$Ùë�$�CE0�01C$�<1$��#�!$;Ï1�657=?< $ A�=?Ï $;�><'� IU(*$;KaÑaÒ!$;C7$]57Ò!$�$�CEC701CKM(*CMÖ»�1Ìr$]=JK��9�65D01CKM01Ó�$:G�=?C7$�Ìn57=?0 �9K4� C7$DÌr(!CE< $�GºG�0.Ña�>ÑR�6CWG �n- 57Ò!$;��57Ò*$ �$�ÑR5701�¾� IUÏ10 CE=¦5EÒ!ÓõÑa=UI?IcG�=U<1$�CEÏ $1-KM0�57Ò*$]Ø�$;KEK7=?� � � � �� � �RÎ,01K7=¦5E=U<1$�G�$râ*�*=¦5E$ A�BÖ�Ìr01(!CWKM$D57Ò!$]Ø$;KEKM=J�6��Ó��.5EC7=Uæº0 Ö�Ó�(!I¦5E=¦LI?�'Ð1$�C�*$r5eÑ40 CEÍ>K=?K:=?� Ï1$��!$;CE� I/�!0 5:Î,01K7=¦5E=U<1$�G!$râ*�!=U57$�$�<1$�CEÐ>ÑaÒ!$�CE$ A,î!0 CD57Ò*$;K7$�CE$;�1KM01�*K57Ò!$,��$;ÑR5701� �6I?Ï 01C7=U57Ò*Ó�=U� =¦5WK@0 CE=UÏ1=U�*� I/ÖÝ01C7Ó�=JK:�!065@(*KE�6P!I?$�ÖÝ0 C@Ï $;�!$�CW�6I��!$�(*CE� I��*$r5MLÑ)01C7Í�I?$;� C7�!=?�!Ï9AlØ0.Ñ)$;< $�C4=U5�Ï =?< $;K4Ï10l0�Gº=U�*K7=?Ï Òl5EK)ÖÝ0 C�G!$�< $;IU01Î!=?�!ÏÙÓ�0 CE$�KM01Î!Ò!=JKe5E=?Ì;�.5E$;G�6I?Ï 0 CE=U57Ò!Ó�K;->�1KaG�=JK7Ì�(*K7K7$;G¾=?�¾5EÒ!$@ÖÝ0 I?I?0.Ña=U�!Ï9A

Page 26: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

ω -½Λ Θ′

-½ΘΛ

U

Network

input

output

ω Networkω

input

output

Θ-½ΛU

Newton Algorithm here ...... ....is like Gradient Descent there

L<M7N ¹PO �*¹ � ��_Esef7o�d;°�seo1_4��oli�se_Wj1ikjl^@q bed�q�_Ebhseik_Wg�d;°�seo1_a�4_E��sed;j�\��¦^;d;bei�seolZ�©

����� � �<3���*�4&�2��%���')� 9 "�%13;�

ÞRÒ!$�CE$�� C7$�K7$�< $;CE� I/=?Ó�Î901CM5W�6�l5�Î!C701Î9$;CM5E=U$�K=?�úÌr0 �Q�e(!Ï1�657$�Ï CW� G�=?$��l5@0 Î�5E=UÓ�=?ë;�.5E=U01� C � ���=¦5Ù=JK�� ���� �@Ó�$r57Ò*0>G%- � E �:=¦5�G�0>$;K7� R 5](9KM$�5EÒ!$�Ø�$�K7K7=?� �ú$ræ�Î!I?=?Ì�=¦5EIUÐ1- ��� �:=¦5��65M5E$�Ó�Î�5EK570�â*�9G#G�$;KEÌr$;�15:G�=UCE$;Ìr57=?0 �*Ka57Ò*�65�57CEк5E0�Ó�=U�*=UÓ��6I?I?Ð3K7Î901=UI/57Ò!$ÙC7$�KM(!IU5D�1ÌWÒ!=U$;< $�Gº=?�#57Ò*$Î!C7$;<>=U01(*K�=U57$;CE�657=?0 �*K;- � � ��=¦5R(*K7$;K4�]I?=U�*$:K7$;� CEÌWÒ%-��6�9G�Ó�01KM54=?Ó�Î901CM5W�6�l57I?Ð -S�#" ��=U54Ñ40 CEÍ�K0 �!I?Ð�ÖÝ0 CaP9�.5EÌWÒ�I?$;� C7�!=?�!Ï9A

ÞRÒ!$�5EÒ!=UCWG�Î!CE0 Î,$�C75eÐ:=?K�KMÒ*0.Ña��=U��îv=?Ï (*C7$ ��&!A �KEKM(!Ó�$)Ñ)$)Î!=?ÌWÍ��DG�$;KEÌr$;�15�G�=UCE$;Ìr57=?0 �%-$ A Ï*A,57Ò!$�Ï1CE�1G�=U$;�l5;-957Ò!$;� Ñ4$�Ó�=?�!=?Ó�=?ë�$�� IU01�!Ͼ�¾I?=U�!$�=?�Á5EÒ!=?K�G!=UCE$;Ìr57=?0 � �»IU=?�!$�K7$;�6CWÌWÒ �rA?>(!P*K7$;Ül(!$;�15EIUÐúÑ4$ºK7Ò!0 (!IJGå57CEÐå570#â9�*Gz� G!=UCE$;Ìr57=?0 �z�6I?0 �*Ï#ÑaÒ!=?ÌWÒz57Ò!$¾Ï CW� G�=?$��l5ÙG�0>$;K�!065�ÌWÒ*� �!Ï $3=¦5WK�G�=?C7$�Ìn57=?0 �/-�P!(�5�Ó�$�CE$�I?Ðz=¦5WK�IU$;�!Ï65EÒ �ÀÌr01�P�e(!Ï1�657$#G�=?C7$�Ìn5E=U01� �n-®P,$;Ì�� (*K7$Ó�0.<>=?�!Ï � IU01�!Ï�57Ò!=JK�G�=UCE$;Ìr57=?0 �zÑa=?I?I)�!0 5�K7Î901=UI�57Ò!$3CE$;K7(!I¦5�06Öa57Ò*$ºÎ!CE$�<>=?0 (*KÙ=¦5E$�CW�.57=?0 �/AÞRÒ!$�$�<10 I?(�57=?0 �¾06Ö�57Ò!$]G�$�K7Ì�$��l5�G�=?C7$�Ìn57=?0 �9K�� $ �.5a=U57$;CE�657=?0 ���º=JKRÏ =?< $;�3�1K

� $ � F� � � � $ ��7 � $ � $ � + � � ���ÑaÒ!$�CE$457Ò!$ÌWÒ*0 =JÌr$R06Ö � $ Ì�� ��P,$�G�01�!$R$�=U57Ò!$;C�� Ì;Ìr0 CWG�=?�!ÏD570]î�IU$�5EÌWÒ!$;C��6�*G�Na$;$�<1$;K � � � �

Page 27: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

first descentdirection gradients

conjugatedirectionL M/N ¹PO �*¹ � ��_Esef7oÙd;°�fWd�j �O�l^�\�se_�^;bM\;�1ik_Wj.s)�1i�be_WfEseikd;jlg�i¦j�\�¢�px_7bhbed;b�gh�1bh°J\;fW_�©

� $ � � �&� � $ � + � �&� � $ �� �&� � $ � + � + � �&� � $ � + � � � E �

0 C �®0 IJ�6Í�� �*G3Na=UP*=U$;C7$

� $ � � � �&� � $ � F � �&� � $ � + �;� + � � � � $ �� �&� � $ � + � + � � � � $ � + � 5 � � � �

ÞRÑ)0�G�=?C7$�Ìn5E=U01�*K�� $ �6�9G � $ � + �6CE$@G!$râ*�!$�G�� KaÌr01�P�e(!Ïl�.57$:=UÖ� + $ � � $ � + � �

=ÀA $ A�Ìr0 �Q�e(!Ï1�657$ºG�=?CE$;Ìn5E=U01�*KÙ� C7$�01CM5EÒ!0 Ï10 �*� I�G�=?C7$�Ìn57=?0 �9K]=?�¼57Ò!$3KMÎ9� Ìr$�0 Ö�6�¼=JG�$��l5E=¦5eÐØ�$;KEK7=?� �]Ó��.57CE=Uæ��ÀKM$;$)î�=UÏ1(!CE$:� (X�nA���$�CEÐ@=UÓ�Î,0 C75E�6�l5%ÖÝ01C�Ì�0 �>< $;C7Ï1$��*Ì�$�=U�ÙP90 57Ò�ÌWÒ*0 =JÌr$;K/=?K

ρ κ−1

ρ κ

κω

L M/N ¹PO �!¹ � ��_Esef7oÙd;°�fWd�j �O�l^�\�se_�^;bM\;�1ik_Wj.s)�1i�be_WfEseikd;jlg�i¦j�\�¢�px_7bhbed;b�gh�1bh°J\;fW_�©

�Ï10l0�GÙIU=?�!$RKM$��6CWÌWÒ]Î!CE0>Ì�$;G�(*C7$1A�î*0 C��Î,$�C7ÖÝ$;Ìr57I?Ð]Ül(*� G�CW�.5E=?Ì�ÖÝ(!�*Ìr57=?0 ��Ña=¦5EÒ �í<'� C7=J�6P*IU$�K

Page 28: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

��Ì�0 �>< $;C7Ï1$��*Ì�$aÑa=¦5EÒ!=U� �ÔKM57$;Î*KRÌ��6�ºP9$@Î!CE0.< $�GcA1î*0 C4�!0 ��LhÜl(*� G!CE�657=JÌaÖÝ(!�*Ìr57=?0 �*K��®01I?� Í�6�*GTNa=?P!=U$;C7$ R K�ÌWÒ!01=?Ì�$3K7$�$;Ó�K�Ó�01C7$3C701P!(*KM5;A�&)0 �Q�e(!Ï1�657$�Ï CW� G�=?$��l5'� � ����Ì��6�T�6IJKM0åP,$<l=?$�Ñ4$;Gì�1K��¼KMÓ��6C75ºÌWÒ!01=?Ì�$¾ÖÝ01CºÌWÒ!0>01K7=?�!Ïå57Ò!$ÁÓ�01Ó�$;�l57(!Ó 57$�CEÓ Íl�*0.Ña�ã=?�x�!$�(*CE� I�!$r5eÑ40 CEÍ�5ECE� =U�!=?�!Ï9A*áO5@Ò*� KP9$;$�� � Î!Î!I?=U$�G#Ña=¦5EÒÁIJ�6CEÏ $]K7(*Ì�Ì�$;KEK�=?�ÁÓÙ(!IU57=ULOI?�'Ð1$�C��!$�5eÑ)01C7Í57CW�6=?�!=U�*Ï�0 ��Î!CE0 P!I?$�Ó�K®5EÒ*�.5R�6CE$aÓ�0>G!$�CW�.57$�K7=?ë�$;G�Ña=U57ÒºCE�657Ò!$;C�IU0.ÑTCE$;G�(!�9G!�6�*Ì�ÐÙ=?��57Ò*$G!�.5W�!A/Þ)Ð>Î!=?Ì;�6I�� Î!Î!I?=?Ì;�.5E=U01�*K�CW�6�!Ï1$�ÖÝCE0 Ó ÖÝ(!�*Ìr57=?0 �ç� Î!Î!CE0'æ>=?Ó��.5E=U01�%-�C701P90 57=JÌ�Ìr0 �l5EC701I� � - �O->57=?Ó�$�LOK7$�CE=?$;KRÎ!CE$;G�=JÌn5E=U01�3� �*G3065EÒ!$�CaCE$;� I,<.� IU(!$�G3Î!C701P!I?$�Ó�KRÑaÒ!$�CE$@Ò!=?Ï Ò#� Ì;Ìr(!CW� Ì�Ð=?KÙÑ4� �15E$;GcA�&)IU$��6CEIUÐ 0 �¼IJ�6CEÏ $�� �*GúC7$�G�(!�*G*�6�l5 �»Ì�I?�1K7K7=Uâ9Ì��657=?0 � �:Î!CE0 P*IU$;Ó�K�Ke5E0�ÌWÒ*� KM57=JÌP*� ÌWÍ>Î!CE0 Î*� Ï1�657=?0 � =JK@Ö»�1Ke5E$�C�A ��IU57Ò*0 (!Ï1Òz�.5M5E$�Ó�Î�5WK]Ò*�'<1$�P,$�$;�zÓ�� G�$�5E0 G�$�â*�!$�Ó�=?�!=¦LP*�.5WÌWÒ!$;K � E " �O-.57Ò!$DÓ��6=?��G�=?KE� G!<'� �l5E�6Ï1$a06Ö%Ìr01�P�e(!Ïl�.57$Ï CW� G�=?$��l5�Ó�$r5EÒ!0�G!K�CE$�Ó��6=?�*K�57Ò*�65=¦5�=JK��åP*�65EÌWÒãÓ�$�57Ò!0�G �ÝÎ*� CM5EIUÐzG!(!$3570 57Ò!$�Î!CE$;Ì�=?K7=U01�çCE$;Ül(!=?C7$;Ó�$��l5EK�=U�TI?=U�!$#K7$;� CEÌWÒÎ!C70�Ì�$;G�(!CE$ �rA����� �=*C�! �"�� �A% � ���&3�� + ! �A���ÞRÒ!$ �D(*� K7=¦L ��$;ÑR5701� �À24î�8 ?"�)Ó�$r57Ò*0>G �%� �4=U57$�CW�.5E=U<1$�I?Ð�Ìr01Ó�Î*(�57$�Ka�6�3$;KM57=?Ó��657$@06Ö�57Ò*$=U�>< $;CEK7$�Ø�$�K7K7=?� �%-<� E �:=?KÙ�6� ���� F �@�6I?Ï 01C7=U57Ò*Ó3-<�#� �:CE$;Ül(!=?C7$�K@IU=?�!$�K7$;� CEÌWÒú�6�*G � � �@=U5Ñ)01C7Í�K40 �!I?Ð�ÖÝ01CaP*�.5WÌWÒ3I?$;� C7�*=U�!Ï9A

ÞRÒ!$Î,01K7=¦5E=U<1$G�$�â*�!=U57$:$;KM57=?Ó��657$�0 Öc57Ò!$:=?�l<1$�CWKM$�Ø$;KEKM=J�6��=JK4G�0 �!$:G�=?C7$�Ìn57I?Ð�Ña=¦5EÒ!0 (!5C7$�Ü1(*=UCE=U�*Ï�Ó��657CE=¦æ3=U�><1$�CWKM=?0 �#�6�9G¾P>Ð30 �!I?Ð3(*KM=?�!Ï�Ï1CE�1G�=U$;�l5�=U��ÖÝ01C7Ó��657=?0 �%A ��I?Ï 0 CE=U57Ò!Ó�=ULÌ��6I?I?Ð#57Ò!=JKÙÌ;�6�åP9$�G!$;KEÌrCE=UP,$;Gú� K:ÖÝ01IUI?0.Ñ�KDC<� � �Dâ9CEKM5Ù�3Î90lKM=U57=?< $�G�$�â*�!=U57$�Ó��.5EC7=Uæ � =?KÌWÒ!01K7$��%-�$1A Ï9A � � �*- � E �)5EÒ!$��35EÒ!$]KM$��6CWÌWÒ¾G�=?CE$;Ìn5E=U01�¾=JK�K7$r5R570

� ��.%� � ����.%� � �&� � �/.%�%�4��� �®�@I?=?�!$K7$;�6CWÌWÒÙ=JK�Î,$�C7ÖÝ0 CEÓ�$;G��6I?0 �!Ï �,- ÑaÒ!=JÌWÒ�Ï =?< $�K�57Ò!$�(*Î,G!�657$RÖÝ01C�57Ò!$Î*�6CW�6Ó�$r5E$�CWK�.5a57=?Ó�$'.

� ��.%� � � �/. F � � F10 ��.%� �S�/.%�45îv=?�*� IUI?Ð � � ��57Ò*$#$;KM57=?Ó��657$#0 ÖD5EÒ!$Á=?�l<1$�CWKM$�Ø�$�K7K7=J�6�T=?K�(*Î,G!�657$�GcAR&)01Ó�Î9�6CE$;Gç570ú57Ò*$��$�ÑR5E0 �º� IUÏ10 CE=¦5EÒ!ÓÚ5EÒ!$ �D(9� K7=¦L ��$;ÑR570 ��� Î!Î!CE01� ÌWÒ�0 �*IUÐ��!$�$�G!K�Ï1CE�1G�=?$��l5�=?��ÖÝ0 CEÓ��.57=?0 �/AÞRÒ!$Ó�01KM5)K7(*Ì�Ì�$;KEKeÖÝ(*I;�D(*�1KM=UL��$�ÑR5E0 ���6I?Ï 01C7=U57Ò*Óà=JK�5EÒ!$24C70.Ð�G�$;��LhîvI?$r5WÌWÒ!$�C7Le8:0 IJG>Ö»�6CEP�L?>Ò*�6�*�!0 �»2Rîv8 ?"��Ó�$�57Ò!0�GcA1ÞRÒ!$a(!Î�G!�.5E$aC7(!I?$4ÖÝ0 C®5EÒ!$a$;KM57=?Ó��.57$a0 Ö*5EÒ!$a=U�><1$�CWKM$aØ�$�K7K7=J�6�=?K

���/.%�<� ���/. F ��� � 7�� + � �� + �

� � � +� + �

F � � + � 7�� � � +� + �

� � � � �ÑaÒ!$�CE$�KM01Ó�$�� P!P!CE$�<>=?�657=?0 �*K4Ò*�'<1$@P9$;$���(*K7$;GºÖÝ0 CR57Ò*$@ÖÝ0 I?IU0.Ña=?�!Ï � � �@< $�Ìn5E0 CWK

� � � �&� � �/.%�;� F � � � � ��. F ���%�� � � ��.%� F � �/. F � �45 � � " �

��IU57Ò!01(!Ï Ò%-®� KÙÓ�$��l57=?0 �!$�G¼�6P,0.< $1-%57Ò*$¾Ìr01Ó�Î!IU$�æ�=¦5eÐå=?KÙ0 �!I?Ð ����ÁF��n-�Ñ)$¾�6CE$ºKe5E=UI?I4C7$�LÜ1(*=UCE$;Gz570¼Ke5E0 CE$3�1� � � Ó��657CE=¦æc-�KM0 57Ò*$�� IUÏ10 CE=¦5EÒ!Ó =JK�01�!I?ÐzÎ!CW� Ìn5E=?Ì;�6I)ÖÝ0 C�K7Ó�� IUI�!$r5eÑ40 CEÍ�K]Ña=U57Òç�*0 ��LOC7$�G�(!�*G*�6�l5]57CW�6=?�!=?�!Ï K7$r5WK�A®Na$�Ìr$��l5EIUмK70 Ó�$º<.�6CE=J�6�l5EKÙ$ræ�=?KM5�57Ò*�65�6=?Ó 570�CE$;G�(9Ìr$�Ke5E0 CW�6Ï1$DCE$;Ül(!=?C7$;Ó�$��l5EKG�»K7$�$�$1A Ï9A � � �U�rA

Page 29: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

��� � � �!*� � � �A% � ���&3 �!3 9 @B%(#&%13 � %(')4 � �2' �*��2' 9 �

8@�6(*KEKeL ��$;ÑR5701�ú� �*G¼"%$;< $;�lP,$�CEÏ#QÁ�6CWÜl(*�6CWG>5Ù� IUÏ10 CE=¦5EÒ!Ó �%� �@(*K7$�5EÒ!$ºKEÜ1(9�6CE$ �1�1Ìr0 P*=�6Î!Î!CE0'æ�=UÓ��657=?0 �%-�� E �v� C7$4Ó��6=?�!I?Ð]G�$;K7=UÏ1�!$;G]ÖÝ0 C�P*�65EÌWÒ�IU$��6CE�!=U�*Ï*-$��� �/Ò9�'< $)�@Ì�0 Ó�Î!I?$ræ�=¦5eÐ06Ö ��#�ÁV �D�6�9G � � ��Ó�01KM5@=UÓ�Î,0 C75E�6�l5�-*5EÒ!$�Ð#Ñ40 CE;01�!I?Ð�ÖÝ0 C:Ó�$��6�åK7Ül(*� C7$�G�$�CEC701CI?01KEKÖÝ(!�*Ìn5E=U01�*K;A,ÞRÒ!$�8@�6(*KEKML��$�ÑR5701�Á�6I?Ï 01C7=U57Ò!Óï=JKI?=UÍ1$]5EÒ!$��$�ÑR5701� �6I?Ï 01C7=U57Ò*Ó3-9Ò!0.Ñ4$�< $;C57Ò!$]Ø$;KEKM=J�6��=JK��6Î!Î*C70'æ�=?Ó��657$�G�P>Ð�5EÒ!$ÙKEÜ1(9�6CE$D0 Öv5EÒ!$�l� Ìr01P!=J�6� �ÀKM$;$]�6IJKM0�K7$;Ìr57=?0 � (>A EÖÝ0 C���ÖÝ(!C757Ò!$;CG�=JK7Ì�(*K7K7=?0 � �

��� � �J �

� ��� � � � ��%�

+ � ��� � � � ���� � � + � �&� � �-5 � � & �

ÞRÒ!$:"/$�<1$��>P9$;C7Ï�QÁ�6CWÜl(*�6CWG>5�Ó�$r5EÒ!0�G�=JK4IU=?Í $57Ò!$�8@�6(*KEKML��$�ÑR5701�º�6P,0.< $1- P!(!54=U54Ò9� K4�C7$;Ï (!IJ�6CE=Uë��.5E=U01�ÙÎ*� CE� Ó�$�57$;C �¾5EÒ*�.5)Î!C7$;< $;�15WK®=¦5�ÖÝC701Ó�P!I?0.Ña=?�!Ï@(!Î%- =UÖ%KM01Ó�$a$�=?Ï $��><.�6I?(!$�K�6CE$@K7Ó��6I?I

��� � �J �

� ��� � � � ����

+ � ��� � � � ���� 7���� � � + � �&� � �- � � (X�

ÑaÒ!$�CE$ � G�$��*0657$�K#5EÒ!$ã(*�!=¦5eÐ Ó��.5EC7=UæcADÞRÒ!$x8@�6(9K7K ��$�ÑR5E0 �ÚÓ�$�57Ò!0�GÚ=JKÁ<.� IU=JGßÖÝ01CÜ1(9� G�CW�.5E=?Ì:Ìr0lKe54ÖÝ(!�*Ìr57=?0 �*KaÒ*0.Ñ)$;< $�C4��KM=?Ó�=UIJ�6CRÎ!CE0�Ìr$;G!(!C7$@�6IJK70�Ñ40 CEÍ�K�Ña=U57Ò�H@(!I?IUP*�1ÌWÍlL"%$�=?P!I?$�C�Ì�01KM5a� �*G3=?KaÌ;�6I?IU$�G��D�.5E(!CE� I/8:CE�1G�=?$��l5 �»K7$�$�$ A Ï*A � � - � � - E �*�nA� �Æ*Ë7Ê� ���Å>Ç Ê�Ç � �DÉ�Å>òíÅ���ò���ò��1�>Ëeó�Ä ËMÄ �WÇ�Æ ��ó�Å!ËeÇ)Ä ËMÄ

�ÚÉDøhÅ!ËMøMó���ò%ÆzÄòcÅ6úÇ�Æ� ��

ä $)Ña=UI?I1�*0.ÑúG!=?KEÌr(*KEK/K7$�<1$�CW�6I.5E$;ÌWÒ!�*=?Ül(!$�Kv� =UÓ�$;G��65�Ì�0 Ó�Î!(�5E=U�!ÏÖÝ(!I?I101C/Î*� CM5E=?� I1Ø�$�K7K7=J�6�=U��ÖÝ01C7Ó��657=?0 �3PlÐ �»� ��â9�!=¦5E$ÙG�=/V,$;C7$;�*Ìr$@Ó�$r5EÒ!0�Gc- �»P �RKEÜ1(9�6CE$ �l� Ì�0 P!=J�6�3�6Î!Î*C70'æ�=?Ó��657=?0 ��×ÖÝ0 C38@�6(*KEKeL ��$;ÑR5701�ì� �*Gì"%$;< $��>P,$�CEÏ6LhQÁ�6CWÜl(*�6CWG>5��6I?Ï 0 CE=U57Ò!Ó&�n- �»ÌD��Ìr0 Ó�Î!(!5E�.5E=U01�ì06Ö57Ò!$�G�=?� Ï 01�*�6I%0 Ö®57Ò!$�Ø�$;KEK7=?� �Á�6�*G �ÀG �aP>Ð30 P�5W�6=?�!=?�!Ϻ��Î!CE0�G�(*Ìr5D06Ö®5EÒ!$�Ø$;KEKM=J�6�Á� �*G��< $;Ìr5701CaÑa=¦5EÒ!0 (�5:Ìr0 Ó�Î!(!57=?�!Ï�57Ò!$]Ø$;KEKM=J�6�/A,B57Ò!$;CK7$�Ó�=¦Lh�6�9�6I?Ð15E=?Ì;�6I�5E$;ÌWÒ!�!=JÜl(!$;Ka57Ò*�65�6I?IU0.Ñß5EÒ!$�Ìr01Ó�Î*(�5E�657=?0 �å06Ö�5EÒ!$�ÖÝ(!I?I�Ø�$�K7K7=?� � � C7$�0 Ó�=U5M57$�G P,$;Ì;�6(*K7$�57Ò!$;Ð �6CE$�CE�657Ò!$;CÌr0 Ó�Î!I?=JÌ��.5E$;G3�6�*G�� I?K70�CE$;Ül(!=?CE$:Ó�� �lÐ�ÖÝ0 CEÑ4� CEG 2.P*� ÌWÍ>ÑR�6CWG�Î!C701Î*�6Ïl�.5E=U01�ºKe5E$�Î*K � "!- * �OA

���� !�"�3C"0��% � "�B%(')%(3�� %ä $]Ì;�6�3ÑaC7=U57$@5EÒ!$ �lL[57Ò3IU=?�!$�06Ö�57Ò*$]Ø�$;KEK7=?� �

� $ � � � � � �&� � �%���� $

�� �&� � 7 � � $ � F � �&� � �

ÑaÒ!$�CE$ � $ � � � � � 65D565-D�X65D5654 � ��=?KD��< $�Ìn5E0 C�0 Ö�ë�$�CE01K�6�*G#01�!I?о01�!$&�Ù�6557Ò*$ÙÍlLÀ5EÒÎ90lKM=U57=?0 �%A�ÞRÒ!=JKDÌ;�6�ÁP,$Ù=?Ó�Î!I?$�Ó�$��l57$�GÁÑa=U57Òå��KM=?Ó�Î!I?$ÙCE$;Ìr=?Î,$XC�� � �DÌr01Ó�Î!(�57$Ù57Ò!$Ù570 5E�6IÏ CW� G�=?$��l5)P>Ð�Ó�(!I¦5E=UÎ*IU$:ÖÝ0 CEÑR�6CWG��6�*GºP*� ÌWÍ>ÑR�6CWG�Î!CE0 Î*� Ï1�.5E=U01��Ke5E$�Î*K;A � E � �DG!G �@570�57Ò*$�lLÀ5EÒ Î9�6CW�6Ó�$r5E$�C��6�*GåÌr01Ó�Î!(�57$�� Ï1�6=?� 57Ò!$�Ï CW� G!=U$;�15�-c�6�*G â*�*� IUI?Ð���� �@KM(*P�57CW� Ìr5@P,065EÒC7$�KM(!IU5EK4�6�9G�G�=?<l=JG�$DPlÐ �>A>Û(*$5E0Ù�>(!Ó�$�CE=?Ì;�6I9$�CEC701CEK�=?��5EÒ!=JK)Ì�0 Ó�Î!(�5W�.57=?0 �¾K7ÌWÒ!$;Ó�$�57Ò*$C7$�KM(!IU57=?�!ÏåØ�$;KEK7=?� �zÓ�=UÏ1Ò15��!065�P,$¾Î,$�C7ÖÝ$;Ìr57I?ÐúK7Ð>Ó�Ó�$�57CE=?Ì Aváh�ç57Ò!=JK�Ì��1KM$�=U5�KMÒ!01(!IJGzP,$ ��� � � �[ünÿ/T �Eþ@�1KaG�$;KEÌrCE=?P9$�G¾P,$�I?0.Ñ�A

Page 30: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

���� � �*C�!'-%����!� � � "0�:3 ��� ��')� ��"� �!��"0�<3 > �&'A����% � �!*� � � �A% � ���<3 �:3 9@ % #<%13 � %('-4 � � �2' �*C�2' 9 �=�!.04<�&'-"0����

�KEKM(!Ó�=?�!Ï���Ó�$;�6��KEÜ1(9�6CE$;GºÌ�01KM5aÖÝ(!�*Ìr57=?0 �

�&� � � � �E J �� � � F ��� � � � �;� +1� � � F ��� � � � �;� � � * �

57Ò!$;�¾5EÒ!$�Ï CW� G!=U$;�15a=JK� �&� � ���� � F J �

� � � F ��� � � � �%� + � ��� � � � ��%� � � - �

�6�*Gº57Ò!$]Ø$;KEKM=J�6�¾ÖÝ0 I?I?0.Ñ�Ka� K

� � � � � J �� ��� � � � �

���+ � ��� � � � �

��� 7 J �� � � F ��� � � � �;� + � F���� � � � �

��� ��� 5 �)" � �� K7=?Ó�Î*IU=UÖÝÐ>=U�!Ï�� Î!Î!CE0'æ>=?Ó��.5E=U01��06Öv57Ò!$�Ø�$�K7K7=J�6�º=JK457Ò*$@KEÜl(*�6CE$:06Ö/5EÒ!$'�1�1Ìr01P!=?� ��ÑaÒ!=JÌWÒ=?K���Î,01K7=¦5E=U<1$�KM$;Ó�=ULhG�$râ*�*=¦5E$�Ó��657CE=¦æ¾06Ö�G!=UÓ�$��9KM=?0 � C%� �

� � � � � J �� ��� � � � �

���+ � ��� � � � �

��� �)"$���

ÑaÒ!$�CE$�5EÒ!$aKM$�Ìr01�*G�5E$�CEÓ ÖÝC701Ó A�Ü,A8�#" � �cÑR� K®G�CE0 Î!Î,$;G%A'ÞRÒ*=?K®=JK�$;Ül(!=?<'� IU$;�l5v5E0@�1K7K7(!Ó�=?�!Ï57Ò*�65�57Ò!$��!$r5eÑ40 CEÍ�=JK��I?=?�!$;� CaÖÝ(!�*Ìr57=?0 �Á0 Ö�5EÒ!$]Î*� CE� Ó�$r57$;CEK � A �Ï1�6=?��57Ò*=?K=?KC7$�� G�=?IUÐ=UÓ�Î!I?$�Ó�$��l5E$;G#ÖÝ01C�5EÒ!$ �>LÀ5EÒ Ìr01IU(!Ó��Á0 Ö�57Ò!$ �1�1Ìr01P!=?� � C!ÖÝ01CD� IUIv57CW�6=?�!=?�!Ï�Î*�65M57$;C7�9K�-�� ���Ñ)$RÖÝ01C7ÑR�6CWGÙÎ*C701Î*�6Ïl�.57$1-'57Ò!$;��� E �®KM$�5�57Ò*$� Ìr57=?<l=U5eÐ�06Ö,57Ò!$�01(�57Î*(�5)(!�*=¦5WK�570 � �6�*G�0 �*IUÐ57Ò!$ �lLÀ5EÒ¼0 (�5EÎ!(�5]5E0��1- ��� �]��P*�1ÌWÍlÎ*C701Î*�6Ïl�.57=?0 �úKM57$�μ=JK�5W�6Í1$��z�6�9Gå57Ò!$�Ï1CE�1G�=?$��l5]=?K� Ì�Ì�(!ÓÙ(*I?�657$;G%A

���� + �!� � �C')� ���:4&�2��"03�4 �%(� �&3 9 9 %(')" #!�!��" #<%1

"%$r5�(9K�Ìr0 �9KM=JG�$�C��:Ó�(!I¦5E=¦LOIJ�'Ð $�C�K7Ð�Ke5E$�ÓÚÑa=U57Ò�K70 Ó�$4ÖÝ(!�*Ìn5E=U01�*�6I*P!IU0�ÌWÍ�K�Ña=¦5EÒ � , =?�!Î!(�5WK�-� � 0 (�5EÎ!(�5EKa� �*G �÷Î*� CE� Ó�$�57$;CEK)06Ö%5EÒ!$:ÖÝ0 CEÓ �� � ��� �� �rA2��0.Ñ �1K7K7(!Ó�$DÑ4$:Íl�*$�Ñ� FD� � � @F -cÑaÒ!=?ÌWÒå=JK�� � � � � � Ó��.57CE=Uæ�A%ÞRÒ*$��å=¦5]=JK�Ke5ECE� =UÏ1Òl5DÖÝ01C7ÑR�6CWG�570#Ì�0 Ó�Î!(�5E$57Ò!=JKaÓ��.5EC7=Uæ

� F6�� � F � �

� � + � F6�� F

� � � 7 � �

� � F � � F 5 �)" E �

ä $RÌ�� �]G�C701Î�57Ò*$)K7$;Ìr01�*G@5E$�CEÓ =?� A)Ü9A8�#" E �/�6�*G@5EÒ!$�CE$;K7(!IU57=?�!Ï$�Ke5E=UÓ��657$�0 Ö�57Ò!$RØ�$�K7K7=J�6�=?K)Î90lKM=U57=?< $@K7$�Ó�=¦LhG�$�â*�!=U57$ A�� ÖÝ(!CM5EÒ!$�CRCE$;G�(9Ìn57=?0 �º=?K4� ÌWÒ!=?$�<1$;Gc-l=UÖ/Ñ)$D=UÏ1�!0 CE$� IUI,P!(�5R57Ò*$G�=?� Ï 01�*�6I,57$;C7Ó�KR0 Ö'��� �

��� �C

� FD���� F, � J

$

� FD���� F $

��� $��� , � F 5 �)"�� ��ßKM=?Ó�=UIJ�6C�G�$;C7=?<.�.5E=U01�3Ì;�6�3P9$]G�01�!$:570�01P�5E� =U�¾57Ò!$ � , 57=?Ó�$;K � , Ó��657CE=¦æ � FD� � ��� F6A

Page 31: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

�� � + �!� � �C')� ���:4&�2��"03�4 ����% 9 "0�!4<�<3C�!.��A%1 � �"0�:3 "�3 3C%1*C'-�:. 3�%(��

24�1ÌWÍ>Î!C701Î*�6Ïl�.5E=U01�]Î!CE0>Ì�$;G�(*C7$�K®ÖÝ0 C�Ì�0 Ó�Î!(�5E=U�*Ï�57Ò!$DG�=?� Ï 01�*�6I!Ø$;KEKM=J�6���6CE$aÑ4$�I?I*Í>�!0.Ña�� ��*!- � -4� - �[A9áO5D=JK� KEK7(!Ó�$;G357Ò9�.5D$�� ÌWÒ#IJ�'Ð $;Ca=U�#5EÒ!$Ù�*$r5eÑ40 CEͺÒ*�1Ka57Ò*$]ÖÝ(!�*Ìr57=?0 �*� I%ÖÝ01C7Ó� , �,��� � , ��,����� � , � � � �»K7$�$Ùî�=?Ï (!CE$ � *�ÖÝ01C�5EÒ!$�K7=UÏ1Ó�0 =JG!�6I%�*$r5eÑ40 CEÍ$�rA�K7=?�!Ï�57Ò*$8@�6(*KEKeL ��$;ÑR5701�Ù�6Î!Î*C70'æ�=?Ó��657=?0 � �ÀG�CE0 Î!Î!=?�!Ï57Ò!$457$;C7Ó 5EÒ*�.5�Ìr01�l5E�6=?� � � � � � �%�/Ñ4$)01P�5E� =U� C

� F6���� F$

� � F4���� F $

�)��� � � $ �%� F �)" � �� F6���� F$ , � � FD�

�%� F$� F, �)" " �

�6�*G� F6���� F, J

$

� F6���� F$

� F$ , 5 �)"�& �äx=¦5EÒ1�úP,$�=?�!Ͼ�38@�6(*KEK7=?� �Á�!0 �*IU=?�!$;� C7=U5eÐ�� K:KMÒ!0.Ña� =U�åîv=?Ï (*C7$ � *�ÖÝ01C5EÒ!$�N24îì�*$r5MLÑ)01C7Í�K4Ñ)$@01P�5E� =U�

� F ���� F$ , � � F �

��� F$� � , F � $ , � F �)" (X�

�6�*G� FD���� F, � J

$

� F4���� F$

� � , F � $ , � F 5 �)"�* �ÞRÒ!$�Ìr01KM5@06Ö4Ì�0 Ó�Î!(�5E=U�*Ϻ5EÒ!$�G�=?� Ï 01�*�6I�K7$;Ìr01�*G G!$�CE=U<.�.5E=U<1$;KDP>ÐÁC7(*�!�!=?�!Ϻ5EÒ!$;K7$�$;Ül(*�6L57=?0 �*K@ÖÝCE0 Ó 57Ò!$�IJ� KM5]IJ�'Ð $�C:5E0�57Ò!$�â9CEKM5�0 �!$�=JK�$;KEK7$��l57=J�6I?IUÐÁ57Ò!$�KE�6Ó�$�� K:5EÒ!$�CE$�Ï1(!I?� CP*� ÌWÍ>Î!CE0 Î*�657=?0 ��Î9� KEKR(*KM$�G�ÖÝ01CR57Ò!$�Ï1CE�1G�=?$��l5;-�$�æ�Ì�$�Î�5R5EÒ*�.5a5EÒ!$]K7Ül(*� C7$:06Öv57Ò!$�Ñ4$�=?Ï Òl5WK�6CE$�(*KM$�G#=?�#57Ò!$ÙÑ)$;=UÏ1Ò15E$;G#K7(!Ó�K�A�ÞRÒ!=JKa5E$;ÌWÒ!�!=JÜl(!$]=JKD� Î!Î!I?=U$�G�=U�Á57Ò!$ èM0 Î!57=?Ó�� I/P!CW�6=?�G!�6Ó��6Ï1$;ê�Î!CE(!�!=?�!Ï�Î!C70�Ì�$;G�(!CE$ �»K7$�$ � E ���*�nA

||ω−x ||2y

x

f( )

ωx

x

ω

y

z

1/2

L<M7N ¹PO �*¹ �®\;fM�6q1bed�ql\;^�\�seikjl^@seo1_a�1i¦\;^;d�jl\;� � _Wghghi¦\;j*¯>ghik^�Z�d;iU� g � �¦_7°?s ��\;jl��m4����g � bei¦^;o.s �7©

Page 32: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

���� � � ��*C��"�3C4 ����%���'-� 9 *��6�,�<>�����% �A%1 � �"0�:3 �!3 9 � #&%1�6���:'

áh�ÙÓ��6�>Ð�Ó�$r5EÒ!0�G!Kv57Ò*�65®Ó�� Í $�(9KM$406Ö!5EÒ!$aØ�$�K7K7=?� �%-;57Ò!$aØ�$�K7K7=J�6�]=JK®(*K7$;G]$�æ!ÌrI?(*KM=?< $;IUÐ@=?�Î!C70�G�(9Ìn5EK�Ña=¦5EÒ¾��<1$;Ìr570 C�A áh�l5E$�CE$;KM57=?�!Ï I?Ð - 57Ò!$;C7$�=JK4��Ñ4�'Ð�06Ö%Ì�0 Ó�Î!(�5E=U�!Ï�KM(9ÌWÒ�Î!C70�G�(9Ìn5EK��ÿ�������� �RÏ 01=U�!Ï 57Ò!CE0 (*Ï Òz5EÒ!$¾57CE0 (!P!I?$306ÖDÌ�0 Ó�Î!(�5E=U�!Ï 57Ò*$�Ø$;KEKM=J�6�ç=¦5WKM$;I¦ÖeA�ÞRÒ!$¾â*�!=U57$G�=WV�$�CE$��9Ìr$�Ó�$r5EÒ!0�G3Ì;�6�¾ÖÝ(!IUâ*IUI%57Ò!=JKR5E�1KMÍ�ÖÝ0 C��6�#� C7P!=U57CW�6CEÐ�< $�Ìn5701C��

��� ��� � �

���� � 7����:� F � �

���� � � � �)"�- �

(*KM=?�!Ïå0 �!I?Ðú5eÑ40 Ï CW� G!=U$;�15�Ìr01Ó�Î*(�5E�657=?0 �*K'�»�65�Î,0 =?�l5 � �6�9G � 7 ���íCE$;K7Î9$�Ìn57=?< $;IUÐ"�r-ÑaÒ!=?ÌWÒ#Ì;�6�3P9$�CE$;�1G�=UI?Ð�Ìr01Ó�Î!(�57$�G3Ña=¦5EÒ3P9� ÌWÍ>Î!CE0 Î � �z=JKa��KMÓ��6I?I/Ì�0 �*KM5E� �15-�nA

ÞRÒ!=?K)Ó�$r5EÒ!0�GºÌ�� ��P,$:� Î!Î!I?=U$�G�570�Ìr01Ó�Î*(�57$D57Ò!$:Î!CE=U�9Ìr=?Î*�6I,$�=?Ï $;�l<1$;Ìr570 C)�6�*Gº$�=?Ï $��!L<'� IU(*$�06Ö �÷P>Ð�5EÒ!$�Î90.Ñ4$�CRÓ�$r5EÒ!0�GcA*24к=U57$;CE�657=?�!Ï��6�*G3K7$r5M5E=U�*Ï

� �/.�7 � �<� ��� ��.%���� �/.%� � �#& � �

57Ò!$�< $;Ìr5701C�� �/.%��Ña=UI?I�Ì�0 �>< $;C7Ï1$:570�5EÒ!$ÙIJ�6CEÏ $�Ke5�$;=UÏ1$��>< $�Ìn5701Ca06Ö � � �*G ��� ��.%���:5E0�57Ò*$Ìr0 CECE$;K7Î901�*G�=?�!Ïú$�=?Ï $��><.�6I?(!$ � E �*-4� � -4� � �[A1?>$�$ � I?K70 � � � ��ÖÝ0 C3�6�x$�<1$��xÓ�0 CE$Á� Ì;Ìr(!CW�.5E$Ó�$�57Ò!0�G¾57Ò9�.5 � ���RG�0>$;KR�!0 5�(*K7$Dâ9�!=¦5E$]G�=/V,$;C7$;�*Ìr$�K��6�*G � E �4Ò*� KaK7=?Ó�=?IJ�6C�Ìr01Ó�Î!IU$�æ�=¦5eÐ1A

� �xÄó®ø � �lË �¼Ç���Å��ò ��ò��1�>Ëeó�Ä ËMÄ �ÚÉDøhÅ!Ë.øeó���ò/ÆçÄ�òcÅ åÇ�Æ �

áO5�=?K�=U�l57$;C7$�Ke5E=U�*Ï@570](!�9G�$�CWKe5W�6�*G�Ò!0.ÑìK70 Ó�$a06Ö,57Ò!$�57CE=?ÌWÍ�K�K7Ò!0.Ña��Î!C7$;<>=U01(*KMI?ÐÙ=?���*(*$��*Ì�$0 ��57Ò!$DØ�$;KEK7=?� �%-.=ÀA $ A1Ò!0.ÑãG!0l$�K®57Ò*$Ø�$�K7K7=?� ��ÌWÒ9�6�!Ï1$4Ña=U57Ò�� CEÌWÒ*=¦5E$;Ìn5E(!CE$R� �*G�G�$r5W�6=?I?K�06Ö57Ò!$�=UÓ�Î!I?$�Ó�$;�15W�.5E=U01�%A,Þ)Ð>Î!=JÌ��6I?I?Ð -!5EÒ!$�$�=?Ï $��><.�6I?(!$ÙG!=?KM57CE=UP*(�57=?0 �Á0 Ö�57Ò!$�Ø�$�K7K7=?� �ÁI?0l01Í�KIU=?Í $�5EÒ!$]01�!$]KMÍ1$r5WÌWÒ!$;G3=?� î�=UÏ1(!C7$ E � C*��ÖÝ$;ÑÚKMÓ�� IUI%$;=UÏ1$��><.�6I?(!$;K;-!Ó��6�>оÓ�$�G�=?(!Ó 0 �!$�K�6�*G�ÖÝ$;Ñ�< $;C7Ð�IJ�6CEÏ $01�!$;K;A äå$:Ña=UI?I��!0.Ñ��6CEÏ (!$�57Ò*�65)5EÒ!$ �?ý.üÀû ����ÿ¦û ���:5.ý��!��� �Ña=?IUIcÌ;�6(*K7$57Ò!$@5EC701(!P!I?$:=?�357Ò!$@5ECE� =U�*=U�!Ï�Î!CE0�Ìr$�K7K4P,$;Ì�� (*K7$ � E �!- E E �� �!0 �!L[ë;$�CE0�Ó�$;� �¾=?�!Î!(!5EK�01CR�!$�(!CE0 ��KM5E�.5E$;K � EXE �� Ña=?G!$�<'� C7=J�.5E=U01�*K406Ö�57Ò!$]K7$;Ì�0 �*G3G�$;C7=?<.�.57=?< $�K)ÖÝC701Ó�I?�'Ð1$�C4570�IJ�'Ð $;C� Ìr01C7CE$�IJ�.5E=U01�ºP,$r5eÑ4$�$���KM5E�.5E$�<.�6CE=?� P!IU$�K�A

Þ/0ã$ræ�$�Ó�Î!I?=¦ÖÝÐì5EÒ!=JK�-4Ñ)$åKMÒ*0.Ñ 57Ò!$ $�=?Ï $��><.�6I?(!$åG�=?KM57CE=?P!(�57=?0 � 06Ö��¼�*$r5eÑ40 CEÍç5ECE� =U�!$�G0 �çB�&4NõG!�.5W��=U�zî�=?Ï (!CE$ E � Av&)IU$��6CEIUÐ1-c57Ò!$;C7$�=JK]��Ña=JG�$ºK7Î!CE$;� G 06Öa$;=UÏ1$��><.�6I?(!$;K �ÀKM$;$îv=?Ï (*C7$!� - ��� �*GºÑ4$:0 P*K7$�CE< $57Ò*�6545EÒ!$@CE�657=?0�P,$r5eÑ4$�$;�º$1A Ï9A15EÒ!$:â*CWKe5��6�9G�57Ò*$:$;IU$;< $;�15EÒ$�=?Ï $��><.�6I?(!$R=JK��6P,0 (�5 *!A6ÞRÒ!$�I?0 �*ÏD5W�6=?I!06Ö�5EÒ!$a$�=?Ï $;�l<.� IU(!$�G!=?KM57CE=UP*(�57=?0 �@�»K7$�$�î�=?Ï (!CE$ E � �=?K:CW�.57Ò*$�C:Î*�6=?��ÖÝ(!I�P,$;Ì;�6(*K7$Ù5EÒ!$�CE�657=?0¾P,$r5eÑ4$�$;�Á57Ò*$�IJ�6CEÏ $�Ke5@�6�*G KMÓ��6I?I?$;KM5:$;=UÏ1$��><.�6I?(!$Ï =?< $;K:5EÒ!$ºÌ�0 �*G�=U57=?0 �*=U�!Ï�0 Ö45EÒ!$�I?$;�6CE�!=?�!Ï�Î!CE0 P!I?$�Ó�A0� IJ�6CEÏ $�CW�.5E=U0#Ìr01C7CE$;K7Î,0 �*G!KD5E0Á�P!=UÏ�G!=WV�$�CE$��*Ì�$]=?��57Ò!$Ù�6æ�=?KR0 Ö�5EÒ!$]$;IUI?=UÎ9KM01=?G!� I/K7Ò*�6Î,$;G3$�CECE0 CRÖÝ(!�*Ìr57=?0 � C*57Ò!$ÙI?� C7Ï1$�CR57Ò*$CE�657=?0*-957Ò!$�Ó�0 CE$]Ñ)$�â*�*G ��5W� Ìr0 LOK7Ò!$;IUI�KMÒ*� Î9$�GÁÓ�=?�!=UÓ��*-,ÑaÒ*=?ÌWÒ � C7$�$ræ>57CE$�Ó�$;IUÐ#KM57$�$;Î570.ÑR�6CWG!K)57Ò!$]K7Ó��6I?I%�.æ�=?Ka� �*G¾< $�CEÐ �9�65��6I?0 �!Ï�57Ò*$�IU01�!Ï��6æ�=?K;A���!0 57Ò!$;CDÏ1$��!$;CE� IvÌWÒ*� CE�1Ìn57$;C7=JKM57=JÌ�06Ö�5EÒ!$�Ø$;KEKM=J�6� =U� ÓÙ(!IU57=UL[IJ�'Ð $;C�!$�5eÑ)01C7Í�K=?KD57Ò*$KMÎ!CE$;�1GçP,$r5eÑ4$�$;�ãIJ�'Ð $;CEK;A®áh�Tî�=?Ï (!CE$ E ��Ñ4$3CE0 (*Ï Ò!I?ÐzKMÍ1$r5WÌWÒãÒ!0.Ñí57Ò!$ÁK7Ò*� Î9$�06Ö:57Ò*$Ø�$;KEK7=?� �¼<'� C7=?$;K]ÖÝC701Ó÷P,$�=?�!Ï CE�657Ò!$;C �9�65Ù=U�z57Ò!$ºâ*CEKM5�IJ�'Ð $;C�570 P9$;=U�!ÏåÜl(!=¦5E$¾KM57$;$�Îz=?�

Page 33: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

0 100 200 300 400 500 600 700 800−6

−5.5

−5

−4.5

−4

−3.5

−3

−2.5

−2

−1.5

−1

−0.5

0

Eigenvalue order

10Lo

g

Eig

enva

lue

the ratio between the 1st and the 11th eigenvalues is 8

L<M7N ¹PO �*¹ �/ik^�_Ej �;\;�k�1_�ghq�_EfEshbe�lZ�ikj]\R£��U\W�._7b�ghol\�be_n�:��_Wik^�o.seg�j1_Es[��d;be� � ¢;¡�����~r¢;¤����;£���~n���shbM\�ikjl_n��d;j���¢;�@o>\�j>�6��beikshse_Ej��1ik^�i�segW©

57Ò!$3I?�1Ke5�IJ�'Ð $�C�A ÞRÒ!=?K��QV,$�Ìn5EKÙ57Ò*$3I?$;� C7�*=U�!Ï KMÎ,$�$�Gç�6�9GçÌ�� �zÎ!CE0.<>=?G�$3�6�ã=U�!Ï1C7$�G�=?$��l55703$ræ�Î!IJ�6=?�#57Ò!$�K7IU0.ÑßI?$;� C7�!=?�!Ͼ=U� IU0.Ñ4$�CDI?�'Ð1$�CWK�6�9G�57Ò!$�Ö»� KM5 �»K70 Ó�$r5E=UÓ�$�01KEÌr=?IUIJ�.5E=U�!Ï��IU$��6CE�!=U�*Ï�=U��5EÒ!$IJ� KM5�I?�'Ð1$�C�AX�ì5EC7=JÌWÍ]5E0ÙÌr01Ó�Î,$��9K7�657$a5EÒ!=?K)G�=/V,$;C7$;�154K7Ì;�6I?$a06ÖcIU$��6CE�!=?�!Ï�=?K570�(*K7$5EÒ!$@=U�>< $;CEK7$DG!=?� Ï 0 �9�6IcØ�$�K7K7=?� ��5E0�Ì�0 �l57CE0 I,57Ò*$:I?$;� C7�*=U�!Ï�CE�657$��»K7$�$��6IJK70�K7$;Ìn5E=U01�&!A

� � � �Dø ��Ë7Ä�ô 4òvÊ�Ç�ÄÈ��zÆ*È�ò%Æ òcÅ���ÇaÈ?� Å�Ç� ÉøhÅ!Ë7øeó���ò/Æ� òcÅ6úÇ�Æ� ��

2)$�ÖÝ0 CE$ÙÑ4$�Ì�0 �*Ì�$��l57CW�.5E$Ù=?� 5EÒ!=JK:K7$;Ìr57=?0 � 0 � Ò*0.Ñ 57035E� =UI?0 C@KM$�Ìr01�*G#0 CWG�$;C5E$;ÌWÒ!�!=JÜl(!$;KÖÝ0 C�57CW�6=?�!=?�!ÏDIJ�6CEÏ $)�!$r5eÑ40 CEÍ�K�-�I?$r5�(*Kvâ*CWKe5�CE$�Î,$;�65®K70 Ó�$4CE�657Ò!$;C�Î,$;KEK7=UÓ�=JKe5E=?Ì�Ö»� Ìr5EK��6P,0 (!5�6Î!Î!I?Ð>=U�*Ï�ÌrIJ� KEK7=?Ì;�6I9K7$;Ì�0 �*G�0 CWG�$�C)Ó�$�57Ò!0�G!K;A�Þ/$�ÌWÒ!�!=JÜl(!$;K)(*K7=U�!ÏÙÖÝ(!I?I�Ø$;KEKM=J�6�º=U��ÖÝ01C7Ó��6L57=?0 � �À8@�6(9K7K4L ��$;ÑR570 �/-!"%$�<1$��>P,$�CEÏ6LhQÁ�6CWÜl(*�6CWG>5)� �*G¾2Rî�8 ?"�4Ì��6�30 �*IUÐ�� Î!Î!I?Ð�570�< $;C7ÐKMÓ��6I?IR�!$r5eÑ40 CEÍ�K]5ECE� =U�*$;Gz=?�TP*�.5WÌWÒçÓ�0�G�$ -®Ò!0.Ñ4$�<1$�C�57Ò!0lKM$�KMÓ�� IUIR�!$�5eÑ)01C7Í�K��6CE$º�!0 557Ò!$�01�!$;K@5EÒ*�.5]�*$�$;GúK7Î9$;$;G�=?�!Ï#(!Îú5EÒ!$�Ó�01KM5;A/QÁ01KM5ÙKM$�Ìr0 �9G 01CEG�$;C�Ó�$r57Ò*0>G*K&�»Ì�0 �P�e(�LÏ1�.5E$�Ï CW� G�=?$��l5�-%24î�8 ?�-�A;A�A#�:CE$;Ül(!=?CE$���I?=?�!$rLhKM$��6CWÌWÒå� �*GúÌ�� � 5EÒ!$�CE$rÖÝ01C7$��!0 5]P9$º(*KM$�G=U�ú5EÒ!$ºKM570�ÌWÒ*� KM57=JÌ�Ó�0�G�$ A/Q �6�>Ð 0 Ö45EÒ!$�5EC7=JÌWÍ�K�G�=?KEÌr(9K7K7$;GåÎ!CE$�<>=?0 (*K7IUÐå�6Î!Î*IUÐ 0 �!I?ÐÁ5E0P*�.5WÌWÒ#IU$��6CE�!=U�*Ï*A9î*C701Ó 01(!C$�æ�Î9$;C7=?$��*Ì�$ÙÑ4$]Í>�!0.Ñ 57Ò*�65:��Ì;�6CE$rÖÝ(!I?I?к5E(!�!$;G Ke5E0�ÌWÒ*� KM57=JÌÏ CW� G�=?$��l5�G�$;KEÌr$;�15�=JK�Ò*�6CWGç570úP9$��.5�01�ãIJ�6CEÏ $�ÌrIJ� KEKM=Uâ9Ì;�.57=?0 �ãÎ!CE0 P!I?$�Ó�K;A®î*0 C�K7Ó��6I?IU$;CÎ!C701P!I?$�Ó�Ka5EÒ*�.5@C7$�Ül(!=UCE$Ù�1Ì�Ìr(*CE�657$�CE$;�6IULO<'� IU(*$;G�0 (!57Î!(�5WK:IU=?Í $]=?�ÁÖÝ(*�*Ìn5E=U01� �6Î!Î*C70'æ�=?Ó��6L57=?0 �ì0 C�Ì�0 �l57CE0 IRÎ!CE0 P!I?$�Ó�K;-�Ñ)$#K7$�$�5EÒ*�.5ºÌ�0 �P�e(!Ïl�.5E$3Ï1CE�1G�=?$��l5'�»Ña=¦5EÒ��®0 IJ�6ÍlLON�=UP!=?$�CE$

Page 34: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

0 2 4 6 8 10 12 14 1601234567891011121314151617181920

Eigenvalue magnitude

Number of Eigenvalues

Big killers

L<M7N ¹�� �9¹ �/ik^�_Ej �;\;�k�1_�ghq�_EfEshbe�lZ�ikj]\R£��U\W�._7b�ghol\�be_n�:��_Wik^�o.seg�j1_Es[��d;be� � ¢;¡�����~r¢;¤����;£���~n���shbM\�ikjl_n��d;j���¢;�@o>\�j>�6��beikshse_Ej��1ik^�i�segW©

L<M7N ¹�� O>¹ � �l��seik�U\W�._7be_n�D\rbef7o1i�se_WfEse�1be_;¯Wseol_/gh_WfWd;j>���1_Ebeik�;\�seik�'_%ikg�d;°Use_Wj�ghZ]\;�k�k_Eb�ikj��kdn�®_Ebc�¦\W�._EbegW©

Page 35: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

A�Ü,A7� � � �%�)0QV�$�CWK457Ò!$�P,$;KM5DÌ�0 ÓÙP*=U�*�657=?0 �306Ö�KMÎ,$�$�Gc-!CE$�I?=?� P!=?IU=U5eк�6�9G¾K7=UÓ�Î!I?=JÌr=U5eÐ A ?>$;< $;CE� I�.5M5E$�Ó�Î�5WK@(*KM=?�!ÏTèMÓ�=?�!=�P*�.5WÌWÒ!$;KEê�=?�ú�6Î!Î!I?Ð>=U�*Ï3Ì�0 �P�e(!Ïl�.5E$�Ï1CE�1G�=U$;�l55703IJ�6CEÏ $�� �*G C7$�LG�(!�*G!� �l5@Î*C701P!IU$;Ó�KDÒ9�'< $ÙP,$�$;� Ó��1G�$�CE$;Ì�$��l57I?Ð � � (>- E "�- �"���OA � <'� C7=J�6�l5:06Ö4Ì�0 �P�e(*Ï1�.5E$Ï CW� G�=?$��l5a0 Î�5E=UÓ�=?ë;�657=?0 � �»Ì�� IUI?$;G�KEÌ�� IU$�G�&R8 �RK7$�$;Ó�K�=U�l57$;C7$�Ke5E=U�*Ï C�Ò!$;C7$:57Ò*$�IU=?�!$ÙK7$;� CEÌWÒÎ!C70�Ì�$;G�(!CE$@=?KRCE$�Î*I?�1Ìr$;G¾P>к�B�;ÛÚ"/$�<1$��>P9$;C7Ï�QÁ� CEÜl(*� CEG>545eÐ>Î9$]� IUÏ10 CE=¦5EÒ!Ó � E � �OA����� 8 ������-���! ��"0� 9 "��!4<�&3��!. @B%(#&%13 � %(')4 � �2' �*��2' 9 � %(����� 9

Þ/030 P�5W�6=?� �3Ke5E0�ÌWÒ*� KM57=JÌÙ<1$�CWKM=?0 �#0 Ö�5EÒ!$�"%$�<1$��>P,$�CEϺQÁ�6CWÜl(*�6CWG>5@�6I?Ï 0 CE=U57Ò!Ó 5EÒ!$�=JG�$��=?K]570 Ì�0 Ó�Î!(�5E$�57Ò*$ºG�=J�6Ï 01�*�6I)Ø�$;KEK7=?� �ú57Ò!CE0 (!Ï1Ò¼�#C7(*�!�!=?�!ÏÁ$;KM57=?Ó��.5E$�06ÖR57Ò*$¾K7$;Ìr01�*GG�$�CE=U<.�657=?< $@Ña=¦5EÒ#C7$�KMÎ,$;Ìr5R570�$;�1ÌWÒ¾Î9�6CW�6Ó�$r5E$�C�A!ÞRÒ!$�=U�9Ke5W�6�l5E� �!$�01(*KaKM$�Ìr01�*G¾G!$�CE=U<.�.5E=U<1$Ì��6�#P,$Ù01P�5E� =U�*$;G�<>=?��P9� ÌWÍ>Î!CE0 Î*� Ï1�.5E=U01�3�1K�K7Ò!0.Ña�#=U��5EÒ!$]ÖÝ01C7Ó�(!IJ� Ka0 Ö�K7$;Ìr57=?0 � (�A �KKM0>0 ���1K�Ñ4$aÒ*�'< $R5EÒ!01K7$aC7(*�!�!=?�!Ï]$;KM57=?Ó��.5E$;K�Ñ4$�Ì�� ��(*K7$a57Ò!$;Óà5E0ÙÌr01Ó�Î!(�57$=U�*G!=U<>=JG�(*�6IIU$��6CE�!=U�*Ï�CW�.5E$;K)ÖÝ0 Ca$�� ÌWÒ3Î*�6CW�6Ó�$�57$�C

0 $ , � �� � � �� � �� � � 7 � �#&"���

ÑaÒ!$�CE$ � G�$��!0 57$�K�5EÒ!$�Ï I?0 P9�6I*I?$;� C7�*=U�!Ï�CW�.5E$ -l�6�*G � ��� �� � �� � � =JK��ÙC7(!�*�!=U�*Ï]$;KM57=?Ó��.57$06Öc57Ò*$

G�=?� Ï 01�*�6I/K7$;Ì�0 �*G�G�$;C7=?<.�.5E=U<1$�Ña=¦5EÒÁC7$�KMÎ,$;Ìr5�570 � $ , A �ú=?KD��Î*�6CW�6Ó�$�57$�CR5E0�Î!CE$�<1$��l5 0 $ ,ÖÝC701ÓßP!IU0.Ña=?�!Ï:(!Î�=U��Ì�� K7$)57Ò!$aK7$;Ìr01�*GÙG!$�CE=U<.�.5E=U<1$�=JK�KMÓ�� IUI[-'=ÀA $ A.ÑaÒ!$;�]5EÒ!$R0 Î�5E=UÓ�=?ë;�.5E=U01�Ó�0.<1$;K4=?���,�.5�Î*� CM5WKR06Ö/5EÒ!$�$�CECE0 C4ÖÝ(!�*Ìr57=?0 �%A*ÞRÒ*$�C7(!�*�!=U�*Ï�$�Ke5E=UÓ��657$@=?KÌr0 Ó�Î!(!57$;G��1K

� � F6���� F$ , � .�� � ���%� F�� ��� � F6���� F$ , � ��� � 7 � � F6�G���� F$ , �#& E �

ÑaÒ!$�CE$ � =?K3�çK7Ó��6I?I:Ì�0 �*KM5E� �l5�57Ò*�65�G!$r57$;C7Ó�=?�!$;Kº57Ò*$å� Ó�01(!�l5¾0 Ö]Ó�$;Ó�0 CEÐã5EÒ*�.53=?KP9$;=U�!Ͼ(*K7$;GcA,ÞRÒ!$�K7$;Ì�0 �*G#G�$;C7=?<.�.5E=U<1$;KÌ;�6�#P,$�Ì�0 Ó�Î!(�5E$;G#Î!CE=?0 Ca5E0�57CW�6=?�!=?�!Ï�0.<1$�Ca$ A Ï*A��KM(*P*KM$�5D0 Ö�5EÒ!$]5ECE� =U�*=U�!ϾKM$�5;A ?�=U�*Ì�$]5EÒ!$�Ð#ÌWÒ*� �!Ï $�01�!IUÐ3< $;C7Ð�K7IU0.ÑaI?Ð�5EÒ!$�Ð�0 �*IUÐ3�!$�$�G5703P9$�C7$;$;KM57=?Ó��.57$�G#$�<1$�CEÐ3ÖÝ$�ÑÚ$;Î90�ÌWÒ*K;A<�065E$Ù5EÒ*�.5:57Ò!$�� G*G�=¦5E=U01�*�6I�Ì�01KM5D0.< $;CCE$�Ï1(!I?� CP*� ÌWÍ>Î!CE0 Î*� Ï1�657=?0 �#=JKD�*$�Ï I?=?Ï =?P!IU$�� �*G Ì�0 �>< $;C7Ï1$��*Ì�$Ù=JK � �1K:�¾CE(!IU$�0 Ö�57Ò>(!Ó�P �Á�6P,0 (!557Ò!CE$�$:57=?Ó�$;KRÖ»�1Ke5E$�CR5EÒ*�6�#��Ì�� C7$�ÖÝ(!IUI?Ð�5E(!�!$;G�KM570�ÌWÒ*�1Ke5E=?Ì:Ï CW� G!=U$;�15�� IUÏ10 CE=¦5EÒ!Ó�A

áh��îv=?Ï (*C7$ E E �6�*G E �@Ñ)$DKM$;$a57Ò!$DÌr0 �><1$�CEÏ $��9Ìr$R06Ö�57Ò*$DKM570�ÌWÒ*� KM57=JÌaG�=J�6Ï10 �*� I*"%$;< $��!LP9$;C7Ï�QÁ� CEÜl(*� CEG>54Ó�$r57Ò*0>G �#&"� ��ÖÝ01C��]5E0.Ð�$ræ!�6Ó�Î!I?$@Ña=¦5EÒº5eÑ40�G�=/V,$;C7$;�15�K7$r5WKR06ÖvI?$;� C7�!L=U�!Ï�CW�.5E$;K;AlBDP><>=U01(*KMI?Ð�5EÒ!$�$ræ�Î,$�CE=UÓ�$;�154KMÒ!0.Ña��î�=?Ï (!CE$ EXE Ìr01�l5E�6=?�*K®ÖÝ$;Ñ)$;C �*(*Ìr57(*�657=?0 �*K57Ò*� �3=?�#î�=UÏ1(!C7$ E ��G�(*$@570�K7Ó�� IUI?$�CRI?$;� C7�!=?�!Ï�CW�.57$�K�A

����� � � ��*C��"�3C4 ����%���'-"�3C� " � �!.�� "�4&%13;#!�!.�*C%!#&%1�6���:'=�<>����C% �A%1 � �"��!3

áh��57Ò!$aÖÝ01IUI?0.Ña=?�!Ï@Ñ)$�Ï1=U<1$457Ò!CE$�$a5EC7=JÌWÍ�K�ÖÝ0 C4Ìr0 Ó�Î!(!57=?�!Ï@57Ò!$Î!C7=?�*Ì�=UÎ*� I!$�=?Ï $;�!<.�6I?(!$�2 ��$;ÌrL570 C¾06Ö�5EÒ!$ Ø$;KEKM=J�6�xÑa=U57Ò!01(�5¾Ò9�'<l=?�!ϼ5E0ãÌ�0 Ó�Î!(�5E$#57Ò!$úØ�$�K7K7=J�6�x=U5EK7$�IUÖeA4Na$;Ó�$�ÓÙP,$�C57Ò*�65�=?�TK7$;Ìr57=?0 � � A!(�Ñ4$3� I?K70 =?�l57CE0>G!(*Ìr$�Gç� Ó�$�57Ò!0�Gz570ú�6Î!Î!CE0'æ�=UÓ��657$�5EÒ!$�KMÓ��6I?I?$;KM5$�=?Ï $��><1$;Ìn5E0 C�06Ö:57Ò!$ÁØ$;KEKM=J�6� �ÝÑa=U57Ò!01(�5ºÒ*�'<>=?�!Ï 570¼Ìr01Ó�Î*(�57$�5EÒ!$ÁØ�$�K7K7=?� � �Ù57Ò!CE0 (*Ï Ò�'< $�CW�6Ï1=U�*Ï �»K7$�$�� I?K70 � E * �*�nA

Page 36: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Weight space

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10

−20

−15

−10

−5

0

Hessianlargesteigenvalue:

λ = 0.84max

η = 2.38max

epochs

Learningrates:

η0 = 0.12η1 = 0.03η2 = 0.02

MaximumadmissibleLearningrate (batch):

L<M7N ¹�� �*¹ �.sed.f7ol\;gOseikf@� i¦\;^�d;j>\��vy*_W�'_Wj6{�_Ebe^�| � \�b��.�>\rbM� s4\;�k^�d�beikseo1Z�©!pa\�sM\�gh_Es4°?bed�Zõ¢ � \��lgO|ghi¦\�jlg)��i�seo�~W����_E½1\;Z�q1�¦_EgW©>u®o1_j1_Es[��d;be��o>\;g�d�j1_��¦ikj1_n\�b)�lj1iksn}�¢:ikjlql� seg�\;j>�3~�d;�1seq1�1sn}1iÝ© _�©seo be_W_aq>\rbM\;Z�_Ese_7beg � ¢�®_Wik^;o6segW}9~a{li¦\�g �7©

Page 37: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Weight space

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Log MSE (dB)

0 1 2 3 4 5 6 7 8 9 10

−20

−15

−10

−5

0

Hessianlargesteigenvalue:

λ = 0.84max

η = 2.38max

epochs

Learningrates:

MaximumadmissibleLearningrate (batch):

η0 = 0.76η1 = 0.18η2 = 0.12

L<M7N ¹�� �*¹ �.sed.f7ol\;gOseikf@� i¦\;^�d;j>\��vy*_W�'_Wj6{�_Ebe^�| � \�b��.�>\rbM� s4\;�k^�d�beikseo1Z�©!pa\�sM\�gh_Es4°?bed�Zõ¢ � \��lgO|ghi¦\�jlg)��i�seo�~W����_E½1\;Z�q1�¦_EgW©>u®o1_j1_Es[��d;be��o>\;g�d�j1_��¦ikj1_n\�b)�lj1iksn}�¢:ikjlql� seg�\;j>�3~�d;�1seq1�1sn}1iÝ© _�©seo be_W_aq>\rbM\;Z�_Ese_7beg � ¢�®_Wik^;o6segW}9~a{li¦\�g �7©

Page 38: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

� � � %(' � % ���C� 9 ä $�CE$�Î,$;�65a57Ò!$�CE$;K7(!IU5�06Ö®0 (!C�G�=JKEÌr(*KEKM=?0 ��=?�#KM$�Ìn57=?0 � (�A ""C!Ke5W�6C757=?�!ÏÖÝC701Óí��CE� �*G�0 Óí=U�!=U57=J�6I/< $;Ìr5701C ��-!57Ò!$�=U57$;CE�657=?0 �

� .�� � � �� ��� �� � ��� � �

Ña=UI?I*$;< $��l5E(*�6I?IUÐ�Ìr01�l<1$�CEÏ $4570]57Ò!$�Î*C7=?�*Ìr=?Î*� I*$�=?Ï $;�>< $;Ìr5701C�»0 C4�@< $;Ìr5701C�=U��5EÒ!$�Î!CE=?�*Ìr=?Î*�6I$�=?Ï $��9KMÎ*�1Ìr$��)� �*G ��� � � � �Ña=?I?I/Ìr01�l<1$�CEÏ $D570�57Ò*$]Ìr0 CECE$;K7Î901�*G�=?�!Ï�$;=UÏ1$��><.�6I?(!$ � � � -4� � �OA� � ��.0�&' �� ���!3� �"0�<3 ��!065EÒ!$�C@Ó�$r5EÒ!0�GÁÓ��6Í1$;KD(*KM$�06Ö�5EÒ!$�Ö»� Ìn5:57Ò9�.5�KMÓ��6I?I�Î9$;CM5E(!CMLP*�.5E=U01�*KR06Ö�57Ò!$�Ï1CE�1G�=?$��l5R� I?K70�IU$�� Gº570�57Ò*$�Î!C7=?�*Ì�=UÎ*� Ic$�=?Ï $;�l<1$;Ìr570 CR0 Ö �

� .�� � � �� � �

�%�� � 7 � � � � ���� � � � �

� F � ����

� � � � �#& � �ÑaÒ!$�CE$ �#=?K��@K7Ó��6I?I!Ìr0 �9Ke5W�6�l5;A1BD�!$R=U57$;CE�657=?0 ��06Ö*5EÒ!=JK�Î!CE0>Ì�$;G�(*C7$4C7$�Ül(!=UCE$;Kv5eÑ40:ÖÝ0 CEÑ4� CEG�6�*Gº5eÑ)0�P*�1ÌWÍ>Ñ4� CEGºÎ!C701Î*�6Ïl�.5E=U01�ºKe5E$�Î*KaÖÝ0 Ca$;�1ÌWÒ¾Î9�.5M5E$�CE�¾=?�357Ò!$@5ECE� =U�*=U�!Ï�K7$r5�A

� 3�.0"03�% � � � *����2��"��&3 �&>�� ÞRÒ!$�ÖÝ0 I?IU0.Ña=?�!Ï�CE(!I?$ÙÓ��6Í1$;KD(*KM$�06Ö�5EÒ!$�CE(!�!�*=U�!Ï3�'<lL$�CW�6Ï $D570�01P�5E� =U�¾57Ò*$�I?� C7Ï1$;KM54$;=UÏ1$��><.�6I?(!$@06Ö/5EÒ!$]�'< $;CE� Ï $:Ø�$;KEK7=?� �¾<1$�CEÐ�Ö»� KM5

� .�� � � � � F�� � � 7 �� � ���

��� � � 7�� � ��� �� � � � � �� F � �

��� � � � � 5 �#& � �Þ/0�K7(!Ó�Ó�� C7=?ë�$1->57Ò!$�$;=UÏ1$��><.�6I?(!$�2.< $�Ìn5701C4Ì�0 Ó�Î!(�5W�.5E=U01�*K6C� Aa��CE� �*G�0 Óí< $;Ìr5701C4=JK�ÌWÒ!0lKM$;��ÖÝ01C�=U�*=¦5E=?� IU=?ë;�657=?0 �306Ö���-E Aa�6��=?�!Î!(�5Î9�.5M5E$�CE��=?KÎ!C7$�KM$;�l57$;G¾Ña=U57ÒÁG�$�KM=?C7$�G30 (�5EÎ!(�5;-,��ÖÝ0 CEÑ4� CEG3�6�*G3P*�1ÌWÍ>Ñ4� CEG

Î!CE0 Î*� Ï1�.5E=U01�%-�Ke5E$�Î3=?K�Î9$;CMÖÝ01C7Ó�$;G3�6�9G�5EÒ!$�Ï CW� G�=?$��l5WK�� � � �a� C7$@KM570 CE$;G%-�!A ����� �

� � �� � �=?Ka�1G!G�$�Gº5E0�57Ò!$]Ì�(!C7CE$��l5aÑ4$�=?Ï Òl5a< $;Ìr5701C � -

� Aa��ÖÝ01C7ÑR�6CWG��6�*G#P9� ÌWÍ>Ñ4� CEG3Î!CE0 Î9�6Ï1�657=?0 ��Ke5E$�Î =?KÎ9$;CMÖÝ01C7Ó�$;GÁÑa=¦5EÒ#57Ò!$�Î,$�C757(!CEP9$�GÑ)$;=UÏ1Òl5R<1$;Ìr570 Ca� �*G¾57Ò!$�Ï1CE�1G�=U$;�l5EK�� � � � �R� C7$�KM570 CE$;G%-

"�A457Ò!$�G!=WV�$�CE$��*Ì�$ � � �1��� � � � � F � � � �;�@=?K�Ì�0 Ó�Î!(�5E$;Gå�6�*G 57Ò*$�CE(!�!�*=U�!Ï#�'<1$�CW�6Ï $Ù06Ö57Ò!$�$;=UÏ1$��>< $�Ìn5E0 CR=?KR(*Î,G!�657$�Gc-

&!ARÑ)$@I?0>0 Î3ÖÝC701Ó � E �OL-��& ��(*�15E=UIv��C7$�� K70 �*� P!IUÐ�KM5E� P!IU$@CE$;K7(!IU5�=?KR01P�5E� =U�!$�G�ÖÝ01C���-(>A457Ò!$�01Î�57=?Ó��6I%I?$;� C7�*=U�!Ï�CW�.57$@=JK457Ò!$;�#Ï =?< $;�¾� K

0 � � ( � �� � � 5

áh��î�=?Ï (!CE$ E � Ñ4$aKM$;$457Ò!$R$�<10 I?(�57=?0 ��06Ö957Ò!$R$�=?Ï $;�><'� IU(*$4�1K��DÖÝ(!�9Ìn57=?0 ��06Ö957Ò!$R�>(!Ó�P9$;C�06ÖÎ*�.5757$�CE��Î!CE$;K7$��l5E�657=?0 �*K,ÖÝ01Cv�R�!$;(!CW�6I6�!$�5eÑ)01C7ÍD=U�]��Ò*�6�*G!ÑaC7=U5M5E$��]ÌWÒ*� CE�1Ìn57$;C�C7$�Ìr0 Ï1�!=U57=?0 �5E� K7Í�A�áh� Î!CW� Ìr57=JÌr$�Ñ)$�� G*�6Î�5@57Ò*$�I?$;� ÍÁKM=?ë�$�06Ö)57Ò!$�CE(!�!�!=?�!Ï��'< $;CE� Ï $]=?� 01CEG!$�C:5703Ï $�5ÖÝ$�Ñ4$�C��*(*Ìr57(*�657=?0 �*K@�À� K��6IJK70 =?�*G�=JÌ��657$;Gã0 �ç57Ò!$�â9Ï (!CE$ �rA�áh�ã57Ò*$¾â9Ï (!CE$�Ñ)$�KM$;$357Ò*�65�.Ö×57$;C]ÖÝ$;Ñ)$;CÙ5EÒ*�6� � � � Î*�65M5E$�CE�zÎ!CE$;K7$��l5E�657=?0 �*K@5EÒ!$¾Ì�0 CEC7$�Ìn5Ù01CEG!$�C]0 Ö�Ó�� Ï �!=U57(9G�$�ÖÝ01C57Ò!$¾$�=?Ï $��><.�6I?(!$1-%=ÀA $�5EÒ!$�I?$;� C7�!=?�!ÏÁCW�.5E$�=?KÙC7$�� ÌWÒ!$�GcAvî*C701ÓÔ5EÒ!$º$�æ>Î,$�CE=?Ó�$;�l5EK]Ñ4$º� I?K700 P*K7$�CE< $D57Ò*�65a57Ò!$ �9(*Ìn5E(*�.5E=U01�*KR06Ö�57Ò!$]�'<1$�CW�6Ï1$DØ�$�K7K7=J�6�30.< $�C)57CW�6=?�!=?�!Ï��6CE$�KMÓ��6I?I[A

Page 39: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

0 50 100 150 200 250 300 350 400

0

10

20

30

40

50

60

70

80

Number of pattern presentations

eig

enva

lue

γ=0.1 γ=0.03

γ=0.01γ=0.003

L<M7N ¹�� �,¹ �/�'d;�k�1seikd�j#d�°�seo1_]_Wik^�_Wj6�;\;�k�l_Ù\;g:\�°J�1jlfEseikd�j#d�°�seol_]j6�lZ:{�_7b�d;°�q>\rshse_Ebej�q1be_Egh_Wj1|sM\rseikd�jlg@°?d;b]\�gho>\rbe_n�#�®_Wik^�o.s]j1_Es[��d;be�3��i�seo塺�¦\W�._EbegW} �;£ ���;¤3fWd�j1jl_Wf7sei¦d;jlg�\�j>�¼~r¢;�;¤º°Ube_W_ql\�bM\�Z�_Ese_EbegW©lu®ol_RshbM\;ikj1i¦j1^@gh_Es�fWd;jlghikgOseg)d�°®~n�;����o>\�j>�6��beikshse_Ej��1ik^;iksegW©

Page 40: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

áh� î�=UÏ1(!C7$ E "º�6�*G E &�Ñ4$�KM5E� CM5:Ña=¦5EÒÁ5EÒ!$�KE�6Ó�$�=U�!=U57=J�6I�Ì�0 �*G�=U57=?0 �9K�-��6�*G Î9$;CMÖÝ01C7Ó��â!æ�$;G3�>(!ÓÙP,$�C�0 Ö�$;Î90�ÌWÒ*K�Ña=¦5EÒ3I?$;� C7�*=U�!Ï�CW�.5E$;K�Ì�0 Ó�Î!(�5E$;G3P>кÓÙ(*I¦5E=UÎ!I?Ð>=U�*Ï�57Ò!$]Î*C7$�LG�=?Ìr57$�G�I?$;� C7�*=U�!Ï:CW�.57$4P>ÐÙ�:Î!CE$;G�$�â*�!$;G�Ìr01�*KM5E�6�l5�A6&)Ò*0l0lKM=?�!Ï@Ìr01�*Ke5W�6�l5����»=ÀA $ A (*KM=?�!Ï:57Ò*$Î!C7$�G�=JÌn57$�G�0 Î�5E=UÓ�� I/CW�.57$��a�6I?ÑR�'Ð>K�Ï =?< $;K�C7$�KM=JG�(*� I%$�CEC701CEKRÑaÒ*=?ÌWÒ � C7$�<1$�CEоÌ�IU0lKM$@5E0�57Ò*$$�CEC701C�� ÌWÒ!=?$�<1$;G�P>Ð�57Ò!$DP9$�Ke5RÌWÒ!0 =JÌr$�0 Ö�57Ò!$:Ìr0 �9Ke5W�6�l5;A1áh��065EÒ!$�C)Ñ)01CEG!K;-.5EÒ!$ºè7Î!C7$�G�=JÌn57$�G0 Î�5E=UÓ��6IcCW�.5E$;ê�=JKR0 Î�5E=UÓ��6I/$��!01(!Ï Ò%A

0 0.250.50.75 1 1.251.51.75 2 2.252.52.75 3 3.253.53.75 4

0

0.5

1

1.5

2

2.5

LEARNING RATEPREDICTED OPTIMAL LEARNING RATE

ME

AN

SQ

UA

RE

D E

RR

OR

1 epoch

2 epochs

3 epochs4 epochs

5 epochs

L<M7N ¹�� �*¹ � _W\;jzg �.�>\�be_W�z_7bhbed;b�\�g�\�°J�1jlfEseikd�jçd;°�seo1_ºbM\rsei¦d {�_EsÀ�®_W_Wjz�k_n\rbejlikjl^ÁbM\�se_¾\;jl�q be_n�1ikfEse_W� d;q1seikZ]\;���k_n\rbejlikjl^ºbM\�se_�°?d;bÙ\�°J�l�k���ÁfWd;jljl_EfEse_n�Ájl_EsÀ�®d;be� � ��¤;£ �#�;� � ~n� �7©�u®o1_shbM\�ikjlikjl^@gh_7s)fEd�jlghikgOseg�d;°/�;���:ol\;j>�6��bei�shse_Wj��1ik^;iksegW©

Page 41: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 3

0

0.5

1

1.5

2

2.5

LEARNING RATEPREDICTED OPTIMAL LEARNING RATE

ME

AN

SQ

UA

RE

D E

RR

OR

1 epoch

2 epochs

3 epochs4 epochs

5 epochs

L<M7N ¹�� �*¹ � _W\;jzg �.�>\�be_W�z_7bhbed;b�\�g�\�°J�1jlfEseikd�jçd;°�seo1_ºbM\rsei¦d {�_EsÀ�®_W_Wjz�k_n\rbejlikjl^ÁbM\�se_¾\;jl�q be_n�1ikfEse_W�åd�q sei¦Z]\��)�k_n\rbejlikjl^�bM\rse_�°Jd;b�\�gho>\rbe_n� ��_Wik^�o.s�j1_Es[��d;be�Á��i�seoz¡3�¦\n�'_Ebeg � ~W�'¢r£ �~n¡��;¤ �Á����¢ �#£���� �å~n��� �ú~n� �7} ��£ ���;¤ � �kd.fn\�� �ÙfWd;jljl_EfEseikd�jlg�\;jl�ã~r¢;�;¤�°?be_E_ºql\�bM\�Z�_Ese_Ebeg� ghol\�be_n�Ù�®_Wik^�o.seg �7©lu®ol_RshbM\�i¦j1ikjl^�gh_Es�fWd;jlghikgOseg�d;°�~n�;���@ol\;j>�6��bei�shse_Wj�� i¦^;i�segW©

Page 42: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

Â�� � Ë �>ÊcÉ?�1�>ËeÇ�Äïó�ÄÈ �åÇ�ÄÊcø7É?�>ËeÇ�Ä

�Ì�Ì�0 CWG�=?�!ϼ570ã57Ò*$ CE$;Ì�0 Ó�Ó�$��*G!�657=?0 �*KºÓ�$��l57=?0 �!$�G��6P,0.< $1-)�çÎ!CE�1Ìn5E=¦5E=U01�!$�CºÖ»� Ìr=?�!ÏT�ÓÙ(!IU57=UL[IJ�'Ð $;CR�!$�(!CW�6Ic�*$r5a57CW�6=?�!=?�!Ï�Î!CE0 P!I?$�Ó�Ñ40 (*I?G¾Ï 0�57Ò*C701(!Ï Ò¾57Ò!$@ÖÝ01IUI?0.Ña=?�!Ï�KM57$;Î*KDC� KMÒ>(���$@5EÒ!$�$ræ!�6Ó�Î!I?$;K� Ìr$;�15E$�CR5EÒ!$�=U�*Î!(�5<.�6CE=J�6P!I?$;K4P>кK7(!P�57CW� Ìr57=?�!Ï�57Ò!$�Ó�$��6�� �!0 CEÓ��6I?=Uë;$:57Ò!$�=?�!Î!(�5�<.� C7=J�6P!I?$:570���Ke5W�6�*G*�6CWG¾G�$;<>=?�657=?0 �306Ö �� =¦Ö®Î,01KEKM=?P!I?$ -*G�$�Ìr01C7CE$�IJ�.5E$�57Ò*$�=U�!Î*(�5<.�6CE=?� P!I?$;K;A� Î!=JÌWͺ���!$r5eÑ40 CEÍ�Ña=U57Ò357Ò!$]K7=?Ï Ó�0 =JG�ÖÝ(*�*Ìn5E=U01��K7Ò!0.Ña�3=U�3â*Ï1(!CE$ �� KM$�5a57Ò!$@5W�6CEÏ $�54<.�6I?(!$�KaÑa=¦5EÒ!=?�¾5EÒ!$�CW�6�!Ï1$D0 Öv5EÒ!$]KM=?Ï Ó�01=?Gc->5eÐ>Î!=JÌ�� IUI?Ð 7 �@�6�*G¾L � A� =U�*=¦5E=?� IU=?ë�$@5EÒ!$�Ñ)$;=UÏ1Òl5EK4570�CW�6�*G!0 Ó�<.�6I?(!$�Ka� KRÎ!CE$;KEÌrCE=UP,$;G¾P>Ð�� &*AÞRÒ!$@Î!C7$�ÖÝ$�CEC7$�GºÓ�$�57Ò!0�G¾ÖÝ0 CR5ECE� =U�!=?�!Ï�57Ò*$��!$r5eÑ40 CEÍ�KMÒ*0 (!IJGºP,$�Î!=JÌWÍ $�G¾�1K4ÖÝ0 I?IU0.Ñ�KDC

� =¦Ö957Ò!$)57CW�6=?�!=?�!Ï@KM$�5�=JK�I?� C7Ï1$:�ÝÓ�0 CE$)57Ò*� ���ÖÝ$�ÑzÒ>(!�*G!C7$�GÙKE�6Ó�Î!I?$;K �%�6�9G]CE$;G�(!�9G!�6�l5;-�6�*G�=UÖ�57Ò*$]5E�1KMÍ3=JKÌrIJ� KEK7=¦â9Ì;�.5E=U01�%-*(*K7$ÙKM570�ÌWÒ*�1Ke5E=?Ì�Ï1CE�1G�=?$��l5�Ña=¦5EÒ Ì�� C7$�ÖÝ(!I%5E(!�!=?�!Ï*-0 Ca(*K7$:57Ò!$]KM570�ÌWÒ*�1Ke5E=?Ì@G�=J�6Ï10 �*� Ic"%$�<1$��>P,$�CEÏ�QÁ� CEÜl(*� CEG�54Ó�$r5EÒ!0�GcA

� =¦Ö�57Ò*$@57CW�6=?�!=U�*Ï�K7$r5a=JKR�!065a5E0l0�IJ�6CEÏ $ ->01Ca=¦Öv57Ò!$@5W� K7Í�=?KRCE$�Ï1C7$�K7K7=?0 �%-l(*K7$@Ì�0 �P�e(*Ï1�.5E$Ï CW� G�=?$��l5�A

&)I?�1K7K7=?Ì;�6IcK7$;Ìr01�*G>LO0 CWG�$�C4Ó�$r5EÒ!0�G!K��6CE$@=UÓ�Î!CW� Ìr57=JÌ�� Ic=U�#�6I?Ó�01KM5��6I?Ic(*KM$�ÖÝ(!I/Ì;� K7$;K;AÞRÒ!$D�*0 ��LOIU=?�!$��6CRG�Ð>�*�6Ó�=JÌ�K)06Ö�KM570�ÌWÒ*�1Ke5E=?ÌÏ CW� G�=?$��l5RG�$;KEÌr$;�15R=?�ºÓÙ(*I¦5E=¦LOI?�'Ð1$�C4�!$�(*CE� I

�!$r5eÑ40 CEÍ�K�-6Î9�6C757=JÌr(!IJ�6CEIUÐ�� K�=¦5RÎ9$;CM5W�6=?�*K�5E0ÙÏ1$��!$;CE� IU=?ë;�657=?0 �%-.=JK)KM57=?I?I*Ö»� C�ÖÝCE0 Ó P9$;=U�!Ï�Ñ)$;IUI(!�*G�$;CEKM570>0�GcA1QÁ0 CE$a57Ò!$;0 CE$r5E=?Ì;�6I!Ñ40 CEÍ�� �*G�KMÐ�KM57$;Ó��657=JÌa$ræ�Î,$�CE=UÓ�$;�15W�6I9Ñ)01C7Í�=?K��!$�$�G�$;GcA

8 � ��3C� �,.�% 9 4<% �%(3;� ��A "�A � "�A 2:A ��H�A LhN�A6Q¼A6Ï1CE�657$�ÖÝ(!IUI?ÐÙ�1ÌWÍl�*0.ÑaIU$�G�Ï $4ÓÙ(�5E(*�6I$ræ!ÌWÒ*�6�*Ï $:Ï CW�6�l5EK)ÖÝCE0 Ó Û:�:�Ûà�6�*G$� ?!î)A� ò �Eò/Æ!ò/ÄÊ�òC�~�©��©!t)Z]\rbeiש4�)_W�1bM\����k_n\�bej1i¦j1^�ikjºgOshbe�lfEse� be_n��q>\rbM\;Z�_Ese_7bRghq>\�fW_Wg�� jl\�se� bM\;�,beik_WZ]\�jljli¦\�j

^�bM\�� i¦_Ej6sn©�YÀj � i¦fMo>\�_W�v )© � d��W_Ebn} � ikfMo>\;_E�/YM©*��d;bM�1\;j*},\�j>��u®old;Z]\;ga`�_7seghf7o1_�}*_n�1i�sed;begW}������� ��������������������� �"!$#���%&'(�)#�+*,��#" ���-� ����.0/21��-'3��%+� }'��d��k�lZ�_�� }'ql\;^;_R~r¢��.©.u®o1_ � YOu`cbe_WghgW}9~n�����6©

¢6©��©vt)Z]\�beiש �R\rse�1bM\;��^;bM\;�1ik_Wj.s��®d�be�6g�_54�fWik_Wj.se��� i¦j �k_n\�bej1i¦j1^1© ����������768#%�9���':'(�)#� }~W� � ¢��7¯ ¢;¡6~ ; ¢�� � }%~n�;��¤6©

� ©am�©��®\�shsei�seiש*��i�begOsh|c\;jl�@gh_WfWd;j>� |»d�bM�1_EbvZ�_Eseold6� gc°Jd;bv�k_n\�bej1ikjl^1¯.��_Es[��_W_Wj]gOse_W_Wq�_EgOs®� _WghfW_Wj.s\�j>�Ùjl_E��sed;j=< g�Z�_Eseold6�!© �>�������?68#%�9���':'(�)#� }>£1¯k~E£1~�;l~ ��� }c~n�;�'¢.©

£1©��©>�v_Wf7��_Eb)\;j>�A@D©>y*_r ��lj*©�YÀZ�q1bedr�6ikjl^Dseo1_afWd�j6�'_7be^�_Wj1fW_�d;°%{>\�f7�6{ bed�q>\�^'\rsei¦d;j��k_n\�bej1i¦j1^��i�seo¾gh_WfWd�jl�3d6�1_7b�Z�_Eseold��1gW©�YÀj¾pa\n�6i¦�3u,d;�1be_Ese�E�6�.} � _Ed;¿�be_E� � ikj.sed�j*},\�j>�¾u,_Ebhbe_Wj1fW_�6_ �Oj1dn��gh� i×}l_n�1i�sed;begW} *��B#" � �������.�0#B!C'�D2�&E�FGHGI68#��2�J�� 5'(�)#��2�K�-'MLN#"���5� �O/2��%+%����7/P DQ#"#�� }ql\;^;_Wg)¢;�R;.�'�.©!y,\W��be_Wj1fW_ �%be�k{>\��lZÚt�ghghd.fWi¦\�se_WgW}9~n��¤;� ©

¡6© )© � ©.��ikghold;q9© �����S�B��Q����'(TM#��-U��V!$#�M*�'('3�����XWY�� -#-.�2�K'(�(#� ©, ��¦\�be_Wjl�1d;j�`cbe_WghgW}1�)½6°Jd�bM�*}~W����¡6©

� ©�y%©n��d�shsed��*©6�4jl�kikjl_/\��¦^;d;bei�seolZ�g,\;jl�RgOsed.f7ol\;gOseikf�\�qlq bedr½ ikZ]\�seikd;jlgW©;Y[japa\n�6i¦�a�1\;\��*}W_n�1i�sed;bn}Z ��� �K�P��[\�����Q�K��.]�K�^����������,�>��'(T?#��U��`_aE�FHF�bdcA#��U��BDQ#�9d'0'�DQ�A����T�':#�e� �2�-'(�K'(��'3�af } ®\�Z:{ beiU� ^�_�}*~n�;��¤6©�u®ol_a�)_E��sed�j�Y[jlgOsei�se� se_�� _Ebeik_WgW}> ®\;ZD{1bei¦�1^;_a�4j1i¦��_Ebeghi�s[��`%be_EghgW©

Page 43: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

�6©apD©1�!©1�vbed.d�Z�o1_n\��Ù\;jl�ÙpD©ly*dn�®_�© � �1��sei¦�;\�bei¦\�{l�k_R°J�1jlfEseikd�jÙikj.se_Ebeq�d��¦\�seikd;j�\;jl��\;�l\;q seik�'_j1_Es[��d;be�6gW© 68#%�9P� � � /Q1�-'3��%+� }�¢6¯ �'¢.~�;6��¡�¡.}/~W��¤;¤ ©

¤ © � ©*y/©*���1j6seikj1_:\�j>��t©9��© � _Wik^�_Ej>�*©� �d�Z�ql� seikjl^]gh_WfWd;j>��d;bM�1_7b��1_Ebeik�;\�seik�'_Wg4i¦j¾�l_W_W� |�1d;bh��\rbM�3jl_7s[�®d�be�6gW¯,t�be_W�6ik_E�a© ���������2�B�2��H 5'(�(#�2�X#� ���������� �>��'(T?#��U�� }�~W�����6©�u,d\�qlq�_n\rbn©

� © )©!pR\�be�'_Wjº\�j>���1© ��© � d.d6� �.©a�)d;se_:d;j��¦_W\�bejlikj1^ÙbM\rse_:ghfMol_n� �l�k_Wg4°?d;bagOsed.fMo>\�gOsei¦f:d�q1sei�|Z�ik�n\rseikd�j9©>Y[j@m�©.`�©.y*ikqlqlZ]\�jlj*}�� © ��© � d.d6� �.}6\�j>�:pD©6�!©'u,d�� be_Ese�W�.�.};_n�1i�sed;begW} �Y����� ������0�>�����B���� �$!$#� %�'(�(#�>*,��#" ���-� ����.�/Q1�-'3��%0� };�'d;�¦�1Z�_��6}rq>\�^�_Wg%¤;�'¢";.¤��;¤ © � d�be^'\�j:¬�\;� °?|Z]\�jlj*}1�1\;j � \�se_Wd }! ®t}*~n�;� ~;©

~W� ©a¬@©�Y7©.p4iU\�Z]\;j.sM\�bM\�g®\�j>�Ù�!©H@D©'¬a�1jl^1© *,�����P 5� 9S��=68#%O9S#�P���2'\�����S�B��2�>��'(T?#��U�� © � ik�k_E�.}�)_E� @vd;be��}9~W����� ©

~;~�©am�©v���k_Esef7o1_Ebn© *,��H 5'(�( ��L ��'�DQ#"��� #B! Z 9�'(��%+��"'(�)#�� }vf7ol\;q1se_7bÙ¤ © � ¯�`/d;�k� j1d�Z�i¦\;��seikZ�_\��k^�d;bei�seo1Z�gW}lql\;^�_Eg~W¤��R;l~n¤;¤ ©��;d�olj � ik�¦_7��w��6d�jlgW}1�4_7� @vd;be��} gh_WfWd;j>��_W�1i�seikd�j9}*~n�;¤'�.©

~n¢6©��© � _WZ]\;j*} ��©.�vi¦_Ejl_Wj1gOsed.f7��}�\;j>�m©�pRd;�1bege\rsn©>�4_E�1bM\;�6j1_Es[��d;be�6g/\;j>��seol_®{1i¦\;g���;\�bei¦\�jlfW_� ik�¦_EZ�Z]\ © ����������868#%O9���':'(�(#� }l£ � ~��7¯k~ ; ¡;¤6}c~n�;�'¢6©

~W� ©�y%© � d��¦�1gOse_Ei¦j*© � _n\�j�g �.�>\rbe_4d�q seikZ]\;�ki�s[�]ikjÙseo1_4fWd;j.sei¦j6�1d��lgvsei¦Z�_�m�d�{l{1ikjlg � d;j1bed:q bed;|fE_n�1� be_�©*u,_WfMolj1i¦fW\;�>m�_Wq�d;bhs�pRm)�v|À��� �6}.pR_Wq sn©'d�° � \rseol_WZ]\rseikfWgW}6�4j1i¦��_Ebeghi�s[�Dd;°�� d�� seol_Ebej ®\��kik°?d;bej1iU\6}!y,t�}9~W��¤��6©

~E£1© � © � © � d;�k�l{�\;jl�� )©!��© �v\;j�y*d'\;j*© LN'(� � � 68#%�9���':'(�)#�Q��� ¢�j>��_W�*©)��d�o1jlg � d;ql�6ikjlg�)jlik�'_Ebeghi�sÀ�Ù`%be_WghgW}>�®\;��seikZ�d;be_;},~n�;¤��6©

~n¡6©�u4© � © � _Wgh�'_EgD\�j>����©,¬�\�qlq�_Wj*©��4j |»�¦ikj1_��¦_W\�bejlikj1^�q1bed.fW_Wghgh_EgDikj#\rbhseikªlfWi¦\;�vjl_W� bM\;��j1_Esh|��d;be�6gW©/Y[j��1© � ©lu�\W� �¦_7bn}l_n� i�sed;bn} LI'�D2��%��'(�) -�� �M9H9���#"� D2����':#&����������P�>��'(T?#��U�� }��'d���|�1Z�_a¡.~�}>ql\;^;_Wga~n���"; ¢���� © �%�kgh_W�6ik_Ebn}�t�Z�gOse_EbM�l\�Z�}�~n���;� ©

~ � ©am�d�{�_7bhs%t©��'\�fWd�{1gW© Y[jlf7be_n\;gh_n��bM\�se_Wgcd�°>fWd;j6�'_Ebe^�_EjlfW_/seo1bed��1^�o�k_n\�bej1i¦j1^)bM\rse_�\��1\;q1sM\rseikd�j9©����������P�>��'(T?#��U�� }�~�¯ ¢��'¡";.�����6}c~n�;¤�¤6©

~n�6©�t�© � ©r¬RbM\�Z�_Ebc\�j>��t©r�1\;j1^�ikdr��\�jlj1ik|��)i¦j1fW_Wj.se_W�k�kiש ��4�fEi¦_Ej6s,q>\rbM\;�k�k_W�1�k_n\rbejlikjl^)\;�k^;d;bei�seolZ�g°?d;b�j1_W�1bM\��cjl_EsÀ�®d;be�6gW©Y[j�p:©9��©*u,d�� be_Ese�W�.�.}!_n� iksed�bn} �Y����� ���+�K�N����������V� �$!$#� %�'(�)#��*,��#" ���-�-�K��.�/21��-'3��%+���=*,��#" �-����K��.�>#B!C'�D2�0EF�GHG 68#�"!"�������P � }1q>\;^;_Wg�£'�";.£'¤6}!� \;j � \�se_Wd } ®t�}9~W��¤;� © � d�be^'\�j�¬�\;�1°?Z]\;j1j9©

~W¤ ©C@D©ry9_n ��lj9© LN#"���5� ���� -#�Q�J� � �)#��2�2��� '3���Y�H����� 9H9��-���Q'(�K�-��".���_3 -#�2�P�� 5'(�)#�2��� '\� �-����Q�K��.O%�#"����5� �:f ©v`/o1pTseol_Eghi¦gW}1�4j1ik�'_Ebeghi�s��_�`�©l_Es � ©� ��1beik_ � `c\�beikg��4Y �7}9~W��¤��6©

~W� ©C@D©;y9_r ��1j9© � _Ejl_EbM\��¦ik�n\rseikd�jÙ\;jl�:jl_EsÀ�®d�be�D� _Wghik^�j�gOshbM\rse_W^�ik_WgW©�YÀj:m�©'`c°J_Eik°?_Ebn}���©.�6f7o be_Ese_Ebn}��©/�1d�^�_E�¦Z]\�j9}/\�j>�Áy/©v�.se_W_W�kgW}%_n�1i�sed�begW} 68#��2�J�� 5'(�)#��2�K�-% �K�`*Y�����)9Q�� 5'(�K�� }�t�Z�gOse_EbM�l\�Z�}~W��¤;� © �%�¦gh_E� ik_Ebn©9`cbed.fW_W_n� ikjl^�g�d�°!seol_�Y[j.se_Ebejl\�seikd�jl\;�! �d;j1°J_7be_WjlfW_R �d�j1jl_WfEseikd;jlikghZ ikj]`�_Ebh|ghq�_EfEseik�'_�}1�4jlik�'_7beghiksÀ�]d;°�� «�1beikfMo9}*~n�6©Q;�~W� ©!�4f7sed�{�_Eb�~n�;¤�¤6©

¢�� ©C@D©6y9_n ��lj9}.�)© ��d�gh_7bn}l� ©1�!©.pR_Ejl�'_Ebn}�p:© � _Wj>� _Ebeghd�j*}.m© ��© � dr�®\�bM�!} � © � �1{l{l\�bM�*}'\;jl�y%©!p:©*�'\�f7�'_E�Ý© � \;jl� ��bei�shse_Wj¾�1ik^�i�sRbe_WfEd�^�j1i�sei¦d;j¾��i�seo¾\Ù{l\;fM� q bed�ql\;^'\rseikd�j3j1_Es[��d;be��©�Y[jpD©!�!©lu,d;�1be_Esegh�.�.}>_W�1i�sed;bn} �Y����� ���0�K� ���������� � �$!$#� %�'(�(#� *,��#" ���-�-�K��.A/Q1�-'3��%+���?��#���� }�� \;j � \rse_Wd1}� ®t}*~n�;��� © � d;be^'\�j�¬�\;� °JZ]\;j*©

¢.~�©C@D©'y*_r ��lj*}.�1© ��©'pR_Wj1�'_Ebn}�\;jl���!© t©'� d;�k�U\6©9�4q1seikZ]\;� {1bM\�ikj��l\�Z]\;^�_;©�Y[j@pD©.��©'u,d;�1be_Esegh�.�.}_W�1i�sed;bn} ������� ����Y�K���>�����B��2� �"!$#� %��'(�)#�&*��B#" ����-�-�K��.�/Q1�-'3��%+���\��#�� � }'ql\;^;_Wg�¡;�;¤R; ���'¡.}~W���;� ©

¢;¢6©C@D©6y9_n ��lj9}6YM©.¬�\;j.se_Ebn}.\;j>�Ù��© t© �6d��k�¦\ ©%�6_WfWd;j>��d;bM� _Eb®q bed�q�_Ebhseik_Wg®d�°,_7bhbed;b�gh� bh°×\;fE_WgW©cY[j������� ����Y�K� �>�������Q� �"!$#� %��'(�)#� *,��#" ���-�-�K��.&/21��-'3��%+��� ��#����� } �1\�j � \rse_Wd1}1 ®t�}>~n�;� ~;©� d�be^'\�j�¬�\;�1°?Z]\;j1j9©

¢�� ©C@D©1y9_r ��1j9}1`�©S@D©l� ikZ]\�bM�!}l\�j>���)©1`�_n\�be�kZ:� shse_Ebn©%t��1sed�Z]\rseikfR�k_n\�bej1i¦j1^:bM\�se_4Z]\�½ ikZ�i¦�W\�|seikd;jå{.�Ád�j |À�kikjl_�_WgOseikZ]\�seikd�júd�°Rseol_�ol_Wghghi¦\;j < gÙ_Ei¦^;_Wj6�'_WfEsed�begW©zY[j � ik�¦_EgW} � \�jlghd;j9}v\;jl� �dn�®\;j9}9_n� iksed�begW} ������� ��������N�>�����B�� � �$!$#� %�'(�(#�N*��B#" ��� �-�K��.I/21��-'3��%+������#��� � },�1\�j� \rse_Wd1}� ®t}*~n�;��� © � d;be^�\;j�¬�\;� °JZ]\�jlj9©

¢r£1© � © �"! �¦�k_Ebn©ãtÚghfn\;�k_n� fWd�j �O�l^�\�se_�^;bM\;�1ik_Wj.sÙ\;�k^;d;bei�seolZ�°Jd�b]°×\;gOsÙgh�1q�_Ebe�6i¦gh_W� �k_n\�bej1ikjl^1©����������P�>��'(T?#��U�� } � ¯ ¡;¢�¡";6¡;�;� }c~n�;���6©

Page 44: Efficient BackProp Yann LeCuni, Leon Bottoui, Genevieve B. Orr2

¢;¡6© � © �"! �k�¦_7bn© � �1q�_Ebe�6i¦gh_W�Á�k_n\rbejlikjl^ºd�jÁ�¦\�be^;_�be_n� �ljl�l\;j.sshbM\;ikjlikj1^¾gh_EsegW© � �2'3��� �P'(�)#��P��� #��S� �P�\#B!������S�B��=/Q1�-'3��%0� }>£ � ~��7¯k~n¡"; ¢;¡6}c~W�����6©

¢ � ©�� © ��© � d.d6�6��\;jl�� )©��1©rpa\rbe�'_Wj9©;�>\�gOs��k_n\rbejlikjl^�ikjj1_Es[��d;be�6g,d�° �¦d.fW\;�k��� |Ýse�1jl_n�aq1bed.fE_Wghghikjl^�1jli�segW© �����S�B��?68#�%O9���':'(�)#�� },~;¯ ¢;¤ ~ ; ¢��;£1}%~W��¤��6©

¢;�6©a�© � �1bM\�sM\6© _:��� � 9S�J���5�af ©�`%olpTseo1_WghikgW}l�)jlik�'_Ebeghi�sÀ��d�°%u,d;�6�'d1}*~n����¢6©¢�¤ ©a�© � � bM\�sM\6}c¬@© |Àm�© �#«�l�k�k_Ebn}/t© �!ik_Wo1_�}%\�j>�#��©ct)Z]\rbeiש�t4�1\;q sei¦��_�d�j1|»�kikjl_]�k_n\�bej1ikjl^�ikj

fMo>\�jl^;i¦j1^�_Wj6�6i�bed�jlZ�_Ej6segW©úY[j � ikfMo>\;_E�a )© � d��W_Ebn} � ikf7ol\;_W�RYM©v��d;bM�1\;j9}®\;jl�úu®old;Z]\;g`�_7seghf7o1_�}�_n� i�sed;begW} ������� ���� �K� ������������ �"!$#� %��'(�)#� *��B#" ��� �-�K��.I/21��-'3��%+� }%�'d;�¦�1Z�_�� }ql\;^;_¡;���6©�u®o1_ � YOuz`%be_EghgW},~n�;�'�.©

¢�� ©�t�© �D©��4q1q�_Wjlo1_WikZÔ\;jl�úm© � ©�� fMo>\�°?_Ebn©�� � .�K':���/Q� .�P��*,��#" ���-� ����. ©�`%be_Ej6seikfW_ � \��k�Ý}�%jl^;�k_E�®d.d6�� ��ki�¿!gW},~W�'��¡.©

�;� © � ©l�)©>�)bhbn©�� 1��P�%+�) 5����P�0��� .H#� �K'�D�%+� !$#�7/2':#" DS�-'(�) +� ��� �2�K��. ©/`/o1pTseol_WghikgW}��)be_W^�d;j� bM\;�1�l\�se_aY[j1gOsei�se�1se_�}!~n�;�'¡.©

�6~�© � © �)©D�)bhbn© m�_WZ�dr�6i¦j1^¼jld;i¦gh_ ikj d;j1|»�kikjl_ gh_n\�befMo��lghikjl^ç\��1\;q sei¦��_ {l\�sefMo ghik�W_EgW©éY[j� ikfMo>\�_W�v )© � d;�W_Ebn} � ikf7ol\;_W�/Y7©*��d�bM�l\�j9}9\;jl�3u®old;Z]\;ga`�_7seghf7o1_�}*_n�1i�sed;begW} ������P ���&��������������� �"!$#� %��'(�)#�+*��B#" ��� �-�K��.0/21��-'3��%+� }.�'d;�¦�1Z�_�� }�q>\�^�_�¢;��¢6©6u®o1_ � YOu `%be_EghgW}l~n�;�'�.©

��¢6© � © �1© y%©>�)bhbn©*m�_W^��1�¦\�beik�n\�seikd;j�ikj�seol_�gh_W�k_WfEseikd�jÙd;°*bM\�� i¦\;�!{>\�ghi¦gv°J�lj1fEseikd�j]fW_Wj.se_EbegW© ����������68#�%O9���':'(�)#�� }!� � ���7¯ �����R; �'¢;�6}c~n�;�'¡6©

�;� ©���© t�©>`�_W\�be�kZ:� shse_Ebn©v�>\;gOs�_E½1\;f7s)ZD�l��seikql�kikfn\�seikd;j�{.�Ùseo1_aol_Wghghi¦\�j9© �>�������?68#%�9���':'(�)#� }�6¯k~W£.�$;>~ ��� }c~W����£1©

��£1© � © � ©,`cbe_WghgW}��)©�`�©,���¦\;jlj1_Ebh�.}��!©9t©,u,_W�1�'d��kgh�.�.}9\;j>� � ©,u4© �/_Eshse_Ebe�kikjl^ © �O��% ��� �) -��W��� 5� 9��:������� 6�� �=D2�A���'>#B!�/P 5�)���2'(� �? �*��B#-.��%0%+�K��. ©# ®\�Z:{ beiU� ^�_]�4j1ik�'_Ebeghi�s[��`%be_WghgW} ®\�Z:{ beiU� ^�_�} �%jl^��¦\�j>�*}*~n�;¤�¤6©

��¡6©apD©6�1\;\��!}._n�1i�sed�bn© Z ��� �K�P�,[\��� �2�K��.&��� ����������Q����'(T?#�-U��0_aE�FHF�b cA#��U��BDQ#�9I'�'�D2������T��':#�� � �Q� '(��'(�S'3�af ©Áu®o1_��)_E��sed�j#YÀjlgOsei�se�1se_��6_Ebeik_WgW}v ®\;ZD{1bei¦�1^;_��)jlik�'_Ebeghi�s[�3`cbe_WghgW}v ®\;Z@|{ bei¦�1^�_;}9~W����¤6©

��� ©apD©9�1\�\;�¾\�j>�¾�!©*t�©*� d;�¦�¦\6© �c½1\;fEsaghd;�k�1seikd�jº°Jd;b�d;j1|»�kikjl_:�k_n\rbejlikjl^�ikj¾ZD�l��seik�U\W�._7bajl_W� bM\;�j1_Es[��d;be�6gW© *?D�1�-�) -��PWY���"�)��T [\��'('3��� � }!��£1¯ £����'�$;.£��;£'�6}v~W����¡6©

���6© � ©9� d;Z�q�d��kikjlgh�.�.}��D©*��\�be�;\;i×}9\;j>� � © ��©9� _W�1jl^1©D�4j |»�¦ikj1_:�k_n\�bej1ikjl^�d�°�� i¦fMold�sed�Z�ik_WgW¯9\;��|^;d;bei�seolZ�g4\;jl���k_n\rbejlikjl^ÙfW� be�'_WgW©)Y[jº�;| � ©*�4o*}* )©!¬R�®d;j9}!\�j>�º�!©! �old1}!_W�1i�sed;begW} ��������������'(T?#��U������=D2�Y/2':�'(�K�-'(�) -��2L �� DQ�2�( 5�,*Y��� �:9Q�� 5'(�K�� }6ql\;^;_Wg)~W�'¡";l~n�;� ©l� ikjl^�\;q�d�be_�¯ � d�be�¦��6fWik_Wj.sei�ª>f�}!~n�;�'¡.©

�;¤ ©am�© �!©>� � shsed�j9©®t4�1\;q1seikj1^@{li¦\;g�{.��^�bM\�� i¦_Ej6s4�1_EghfW_Wj.sn¯>t�j�ikjlfEbe_EZ�_Wj.sM\;�,��_Ebeghikd�j�d�°�� _W��sM\�|{l\�bh|À�1_E�ksM\6©�YÀj � ik�k�¦i¦\�Z��6�®\�bhsed�� sn}n_n� iksed�bn} *,��#" �-�-���K��.��#a!?'�D2��E���'�D7�7'(�)#�����68#�$!"���-���� ��#��A�Y��'(� �? 5�)��P� �2'3�5�K� � .����� � }lql\;^�_Eg~n�6~�;l~r� � }9�1\�j���d;gh_�}! ®t�}>�;�l���¾~W����¢6© � Y[uz`%be_WghgW©

�;� ©�`�©E��\�j��1_Eb�� Z]\�^;sn© � ikjlikZ�ikge\�seikd�j�Z�_Eseo1d6�1g�°Jd�b,shbM\;ikj1i¦j1^�°J_W_n�6|Ý°Jd;bh�®\�bM�ajl_7s[�®d�be�6gW© ��������������'(T?#��U�� }!� � ~��7¯k~ ;>~�~;}�~n�;�;£ ©

£�� © �© �v\;q1jlik��© �=D2���7'(���-�C#B!�/2':'(��� '(�( �2[V�-����Q�K��."�=D2��#� 1 ©��6q1beikjl^;_Eb��v_Ebe�¦\�^1}.�)_E�^@vd;be��}~W����¡6©

£ ~�© �©���\�qljlik��© /Q':'(�K�-'(�) -��J[\��� �2����. �=D2��#� 1 � ik�¦_7�.}l�)_E� @�d�be�!}!~W����¤6©£'¢6©�t�© � \�ik{�_W�×} u4© � \;jl\;�n\W�®\ } � © � ikj.sed�j*}.¬@©1� o1i¦�;\;j1d1}6\�j>��¬@©6�1© y9\;j1^1©c`%old;jl_WZ�_�be_WfEd�^;|

j1i�sei¦d;j]�lghikjl^aseikZ�_E|À�1_W�¦\W�@j1_W�1bM\���jl_EsÀ�®d�be� gW© ������� �2�B�2��H 5'(�)#��2�7#��X�Y -#���-'(�) 5����/�9Q�-�� D ���P��/2� .�����*,��#" ���-�-�K��. }�tR�l� `%|À���6¯ �'¢�¤R;6�;���6}c~n��¤;� ©

£�� © � © � i¦_E^�_EbeikjlfM��}1t© ¬ad�Z�d6�1\ }1\;jl��u4© � _Wgh��_WgW©/�6sed.fMo>\�gOseikfa� � j>\�Z�ikfWg®d;°��¦_W\�bejlikj1^:��i�seoZ�d;Z�_Wj.se�lZ ikj�jl_E�1bM\;�*jl_7s[�®d�be�6gW© � #��S� �P�\#B!Y*?D�1�-�) 5�O� }>¢��6¯ £;£.¢�¡$;.£;£'�'�.}%~n����£1©

£;£1© � © � © @®\�jl^�\�j>����©ct�Z]\�beiש�u®ol_�_54�fEi¦_EjlfE��\;j>�ºseol_:bed;{l�lgOsej1_Wghgd;°�j>\rse�1bM\���^;bM\;�1ik_Wj.s� _WghfW_Wj.s��k_n\rbejlikjl^Dbe�l�k_�©%Y[j � ikf7ol\;_W�*Y7©1��d�bM�l\�j9} � i¦fMo>\�_W�9�1©l¬a_W\�bejlgW}l\;jl��� \�bM\@t�©1� d;�¦�¦\6}_W�1i�sed;begW} ������P �����K� ����������2� �$!$#� %�'(�)#�� *,��#" ���-� ����.&/21�� '3��%0� }1��d��k�lZ�_a~n�6© u®ol_ � YOu`cbe_WghgW}9~n���;¤ ©