deeplearning.ai

One hidden layerNeural Network

Neural NetworksOverview

Andrew Ng

What is a Neural Network?

!"!#!$

%&

x

w

b

' = )*! + , - = .(') ℒ(-, %)

x

4["]

,["]'["] = 4["]! + ,["] -["] = .('["]) '[#] = 4[#]-["] + ,[#] -[#] = .('[#]) ℒ(-[#], %)

!"!#!$

%&

4[#]

,[#]

deeplearning.ai

One hidden layerNeural Network

Neural NetworkRepresentation

Andrew Ng

Neural Network Representation

!"

!#

!$

%&

deeplearning.ai

One hidden layerNeural Network

Computing aNeural Network’s

Output

Andrew Ng

Neural Network Representation

!"!#!$

%&

' = )!! + +

)!! + +,

!"!#!$

-(') , = %&'

, = -(')

Andrew Ng

! = #!$ + &

#!$ + &'

$($)$*

+(!) ' = ./!

' = +(!)

Neural Network Representation

$($)$*

./

$($)$*

./

Andrew Ng

Neural Network Representation

!"!#!$

%&

'""

'#"

'$"

'("

)"" = +"" ,! + /"["], '"["] = 3()"" )

)#" = +#" ,! + /#

["], '#["] = 3()#" )

)$" = +$" ,! + /$["], '$["] = 3()$" )

)(" = +(" ,! + /(

["], '(["] = 3()(" )

Andrew Ng

Neural Network Representation learningGiven input x:

! " = $ " % + ' "

( " = )(! " )

! , = $ , ( " + ' ,

( , = )(! , )

%"%,%-

./

(""

(,"

(-"

(0"

deeplearning.ai

One hidden layerNeural Network

Vectorizing across multiple examples

Andrew Ng

Vectorizing across multiple examples

!"!#!$

%&

' " = ) " ! + + "

, " = -(' " )' # = ) # , " + + #

, # = -(' # )

Andrew Ng

! " ($) = ' " (($) + * "

+ " ($) = ,(! " $ )! - ($) = ' - + " ($) + * -

+ - ($) = ,(! - $ )

Vectorizing across multiple examplesfor i = 1 to m:

deeplearning.ai

One hidden layerNeural Network

Explanation for vectorized

implementation

Andrew Ng

Justification for vectorized implementation

Andrew Ng

!"!#!$

%&

Recap of vectorizing across multiple examplesfor i = 1 to m

' " ()) = , " !()) + . "

/ " ()) = 0(' " ) )' # ()) = , # / " ()) + . #

/ # ()) = 0(' # ) )…1 = !(") !(#) !(2)

/["](#)A["] = /["](") /["](2)…

6 " = , " 1 + . "

7 " = 0(6 " )6 # = , # 7 " + . #

7 # = 0(6 # )

deeplearning.ai

One hidden layerNeural Network

Activation functions

Andrew Ng

Activation functions

!"!#!$

%&

' " = ) " ! + + "

, " = -(' " )' # = ) # , " + + #

, # = -(' # )

Given x:

Andrew Ng

Pros and cons of activation functionsa

zsigmoid: ! = 1

1 + &'(

z

a

x

a

z

a

deeplearning.ai

One hidden layerNeural Network

Why do you need non-linear

activation functions?

Andrew Ng

Activation function

! " = $ " % + ' "

( " = )["](! " )! . = $ . ( " + ' .

( . = )[.](! . )

Given x:

%"%.%/

01

deeplearning.ai

One hidden layerNeural Network

Derivatives of activation functions

Andrew Ng

Sigmoid activation function

a

z

!(#) = 11 + )*+

Andrew Ng

!(#) = tanh(#)

Tanh activation functiona

z

Andrew Ng

zReLU

a

zLeaky ReLU

a

ReLU and Leaky ReLU

deeplearning.ai

One hidden layerNeural Network

Gradient descent forneural networks

Andrew Ng

Gradient descent for neural networks

Andrew Ng

Formulas for computing derivatives

deeplearning.ai

One hidden layerNeural Network

Backpropagationintuition (Optional)

Andrew Ng

Computing gradients

Logistic regression

! = #$% + '%#'

) = *(!) ℒ(), /)

Andrew Ng

![#] = &[#]' + )[#]'

&["]

)["]

+[#] = ,(![#]) ℒ(+[0], y)![0] = &[0]' + )[0] +[0] = ,(![0])

Neural network gradients&[$]

)[$]

Andrew Ng

!"[$] = !'[$]( ) *

!+[$] = !'[$]

!'[)] = " $ ,!'[$] ∗ .[)]′(z ) )

!"[)] = !'[)]3,

!+[)] = !'[)]

Summary of gradient descent!'[$] = ([$] − 5

Andrew Ng

!"[$] = '[$] − )

!*[$] = !"[$]' + ,

!-[$] = !"[$]

!"[+] = * $ .!"[$] ∗ 0[+]′(z + )

!*[+] = !"[+]5.

!-[+] = !"[+]

!6["] = 7["] − 8

!*["] = 1:!6["]7 $ ,

!-["] = 1:;<. >?:(!6 " , '5A> = 1, BCC<!A:> = DE?C)

!6[$] = * " %!6["] ∗ 0[$]′(Z $ )

!*[$] = 1:!6[$]G%

!-[$] = 1:;<. >?:(!6 $ , '5A> = 1, BCC<!A:> = DE?C)

Summary of gradient descent

deeplearning.ai

One hidden layerNeural Network

Random Initialization

Andrew Ng

What happens if you initialize weights to zero?

!![!]"#

!$[!]"$%&!!

[$]

Andrew Ng

Random initialization

!![!]"#

!$[!]"$%&!!

[$]