Notes from Coursera Deep Learning courses by Andrew Ng

Neural Network and
Deep Learning
datahacker.rs

What is Deep Learning?
datahacker.rs

 Deep Learning is an artificial intelligence function that imitates the
workings of the human brain in processing data and creating patterns for
use in decision making.
 Deep Learning transformed internet businesses like web search and
advertising.
 Deep Learning is enabling brand new products and businesses.
001 – What is Deep Learning?

 Deep Learning uses:
 Medicine (reading x-ray images)
 Self driving car
 Agriculture (precision agriculture)
 And many more...

 Opportunity to create an amazing world powered by AI.
 AI is the new electricity
 Electricity transformed every major industry, from manufacturing to
healthcare and many more.
 AI is on path to bring about an equally big transformation.
 Big part of rapid rise of AI is driven by developments in deep learning.
 One of the highly sought out skills.

 What will we learn?
 Neural Networks and Deep Learning
 You will learn how to build a neural network including a deep neural network.
 How to train it on data.
 In the end, you will be able to build a deep neural network to recognize cats.
 Improving Deep Neural Networks
 Hyperparameter tuning, Regularization, Optimization, how to diagnose bias and variance.
 Structuring your Machine Learning project
 The strategy has changed in the era of deep learning.
 A split of train and test
 End-to-end deep learning
 Convolutional Neural Networks
 Often applied to images
 Natural Language Processing
 Building sequence models
 Applying them to sequence data
 Natural language is just a sequence of words

002 - Neural Network
 Deep learning refers to training neural networks sometimes very large
neural networks.
 A Neural Network, is a technology built to simulate the activity of the
human brain. Specifically, pattern recognition and the passage of input
through various layers of simulated neural connections.
 Inspired by human mind
 Neurons are activated by some input
 Massively used for problems of classification and regression
 Archived amazing results in the field
 Not so new, developed in 1940s, 1950s and 1960s

Speech Recognition
Object Recognition
on Images
Driving Cars (Googles
self-driving car)
House Price
Prediction
Image Processing
(Face Recognition)
Advertisement
(Whether a user
would click on an
advertisement {0, 1 })
Where can neural networks be applied.

Housing Price Prediction.
We have a dataset with six
houses, and we know the size
of these houses in square feet
or square meters and the price.
Now we want to fit a function
to predict the price of a house
as a function of the size, so we
apply a straight line to our
graph.
Since we know that the price can never
be negative, we bent the curve so it ends
up been 0.
Straight line fit

X (size of the house) goes into this node
this little circle (hidden unit) and then
outputs Y, which is the price of the house.
This little circle is a single neuron in a
neural network implementation.
 This is a tiny little neural
network and larger neural
network are formed by taking
many of these single neurons
and stacking them together.

 Now we have know other things about
these house other than just the size
{features}
- number of bed rooms
- zip code & postal code { country }
- wealth { school quality, grocery store }
 The family size, walkability and school
quality helps us predict the price.

 I our e a ple, X are all of these
four i puts a d Y is the pri e e are
trying to predict. So by stacking together
a few of the single neurons we now
have a slightly larger network.
 The magic of a neural network is that
when you implement it, you need to
gi e it just the i puts X a d the
output Y for a u er of e a ples i
your training set and all these things in
the middle, they will figure out by itself.

Supervised Learning with
Neural Networks
datahacker.rs

 There's been a lot of hype about neural networks and perhaps some of that hype is
justified given how well they're working.
 But it turns out that so far almost all the economic value created by a neural networks
has been through one type of machine learning called supervised learning.
003 - Supervised Learning with Neural Networks

 You have some input X and you want to learn the function mapping to some output Y.
 So for example just now we saw the housing price prediction application where you input some features of a home and try to output or estimate the price Y.
 The single most lucrative application of deep learning today is online advertising. Neural networks have gotten very good at predicting whether or not you click
on an ad and by showing you and showing users the ads that you're most likely to click on.
 You might input an image and want to output an index from one to a thousand trying to tell you if this picture it might be any one of a thousand different
images so you might use that for photo tagging.
The recent progress in Speech recognition has also been very exciting where you can now input an audio clip to, a neural network can have it output a text
transcript.
 Machine translation has also made huge strikes thanks to deep learning where now you can have a neural network input an English sentence and directly
output their Chinese sentence.
 In Autonomous driving you might input then say a picture of what's in front of your car as well as some information from a radar and based on that maybe in
your network can be trained to tell you the position of the other cars on the road so this becomes a key component in autonomous driving systems.
Input(x) Output(y) Application
Home features Price Real Estate
Ad, user info Click on ad?(0/1) Online Advertising
Image O je t ,…., Photo tagging
Audio
English
Image, Radar info
Text transcript
Chinese
Position of other cars
Speech recognition
Machine translation
Autonomous driving
Standard Neural Network
Convolutional Neural Network
Custom / Hybrid Neural Network
Recurrent Neural Network

Neural Network examples
Standard Neural Network (SNN) Convolutional Neural Network (CNN) Recurrent Neural Network (RNN)

Structured Data Unstructured Data
 databases of data with a columns.
 each of the features such as size of a house
on upper bedrooms or the age of a user has a
very well-defined meaning.
Size
2104
1600
#bedrooms
3
4
Price(1000$s)
400
540
User Age
41
18
Ad Id
93242
71244
Click
0
1
 refers to things like raw audio, images or text.
 it's been much harder for computers to make sense of
unstructured data compared to structured data.
Text
Audio Image

 Thanks to neural networks computers are now much better and interpreting unstructured data as
well compared to just a few years ago. This creates opportunities for many new exciting applications
that use speech recognition, image recognition, natural language processing on text much more than
it was possible even just two or three years .
 Creating has also been on structured data such as much better advertising systems , much better
private recommendations and just a much better ability to process the giant databases that many
companies have to make accurate predictions from them .
 So Neural Networks have transformed Supervised Learning and are creating tremendous economic
value. It turns out that the basic technical ideas behind Neural Networks have mostly been around
sometimes for many decades , so why is it then that they're only just now taking off and working so
well.

The Rise of Deep Learning
datahacker.rs

 If the basic technical idea behind deep learning neural networks has
been around for decades why are they only now taking off?
 To answer this question, we plot a figure where on the x-axis we plot
the amount of labeled data we have for a task, and on the y-axis we
plot the performance of our learning algorithm (accuracy).
 For example, we want to measure accuracy of our spam classifier or
the accuracy of our neural net for figuring out the position of other
cars for our self-driving car.
004 – The Rise of Deep Learning

 If we plot the performance of a traditional learning algorithms
such as Support Vector Machine or Logistic Regression as a
function we will get the following curve:

 If we plot the performance of a traditional learning algorithms
such as Support Vector Machine or Logistic Regression as a
function we will get the following curve:
For small datasets classification
tasks we get good results
For huge datasets classification
tasks we get performance plateau

 How to overcome performance plateau problem?
Traditional classifiers

 How to overcome performance plateau problem?
In this regime of smaller training sets, the
relative ordering of the algorithms is not very
well defined.
Often, skill at hand engineering features is
what determines performance!
Traditional classifiers

 Over the last 20 years we accumulated a lot more data for lot of
applications than traditional learning algorithms were able to effectively
take advantage of.
 Enormous amount of human activity is now in the digital realm were we
spend so much time on a internet.
 And with neural networks it turns out that if you train a very large neural
network then its performance often keeps getting better and better.

 In the early days in the modern rise of deep learning it was a scale of
data and a scale of computation constraints which limited our ability to
train very large neural networks on either a CPU or GPU.
 But relatively recently there have been tremendous algorithmic
innovations all of them trying to make neural networks run much faster.
 For example, switching from a sigmoid function to a ReLU function has
been one of the huge breakthroughs in neural networks with large
impact on their s performance.

 Let s look i ore detail h s it hi g fro sig oid to ReLU fu tio did
so well on NN performace.
 One of the problems of using sigmoid functions
in machine learning arrises in these regions.
 Here the slope of the function with a gradient is
nearly zero that is learning becomes really slow
when gradient descent is implemented.
 The gradient is zero and the parameters change
very slowly yielding to very slow learning.

 For the Rectified Linear Unit function, the
gradient is equal to 1 for all positive values of
input.
 The gradient is much less likely to gradually
shrink to 0, and the slope of the line on the
left is 0.
 At the end, just by switching from the
sigmoid function to the RELU function has
made an algorithm called gradient descent to
work much faster.

 Often you have an idea for a neural network architecture and you want
to implement it in code.
 Fast computation is important because the process of training neural
network is very iterative and can be time consuming.
 Implementing your idea then lets you run an experiment which tells you
how well your neural network does. Then, by looking at it, you go back to
change the details of your neural network and then you go around this
circle over and over.

Idea
Experiment Code

 Entire deep learning research community has been incredible at just
inventing new algorithms and making nonstop progress on that front.
 These are some of the forces powering the rise of deep learning and the
good news are that these forces are still working powerfully to make
deep learning even better.
 Ability of the very large neural networks from computation point of view
will keep on getting better, as society will continue throwing off more and
more digital data.

Binary Classification
datahacker.rs

 Logistic regression is an algorithm for binary classification
 Example of a binary classification problem:
 you have an input (x) of an image and
 the output (y) is a label to recognize the image, 1 (cat) vs 0 (non cat)
 In binary classification our goal is to learn a classifier that can input
an image represented by its feature vector X and predict whether the
corresponding label y is 1 or 0, that is whether this is a cat image or
a non cat image
005 - Binary Classification
1 (cat) vs 0 (non cat)
y

 Image representation in computer: The computer stores 3 separate matrices
corresponding to the red, green and blue color channels image
 If the input image is 64 by 64 pixel, then you would have three 64 by 64 matrices
corresponding to the red, green and blue pixel intensity values for your image.
 64 by 64 image - the total dimension of this vector X will be: =64*64*3=12288

Notation
 Single training example is represented by , where , label
 Training set m, training examples:
 All training examples: ▪ Output:
 is x dimensional matrix:
 Python command for finding the shape of X matrix is X.shape = (n,m)
 is 𝟏x dimensional matrix:
 Python command for finding the shape of y is y.shape = (1,m)

Logistic Regression
datahacker.rs

 Logistic regression is a supervised learning algorithm that we can use when
the output labels are all either: 0 or 1.
 This is, so called, binary classification problem.
 An input feature vector x may be corresponding to an image that we want
to recognize as either a cat picture (1) or not a cat picture (0).
 That is, we want an algorithm that can output the prediction ( ) which is
an estimate of y:
006 – Logistic Regression

 More formally, we want y to be the probability of the chance that y is equal
to 1, given the input features x.
 In other words, if x is a picture, we want to tell us what is the chance
that this is a cat picture.
 An x is an nx dimensional vector.
 The parameters of logistic regression will be w which is also an nx
dimensional vector together with b which is just a real number.
 Given an input x and the parameters w and b, how do we generate the
output ?
 One thing we could try, that doesn't work, would be to have:

 is a linear function of the input x and in fact this is what we
use if we were doing linear regression.
 However, this isn't a very good algorithm for binary classification, because
we want to be the chance that y is equal to 1, so should be between
0 and 1.
 It is difficult to enforce, because can be much bigger than 1 or can
even be negative which doesn't make sense for a probability that we want
to be between 0 and 1.
 So, in logistic regression our output is going to be the Sigmoid function.

Sigmoid Function of z
 It goes smoothly from 0 up to 1
Sigmoid Function in Python

 We are going to use z to denote this quantity
 Then, we have:
 If z is large:
 If z is large negative number:
 When we implement logistic regression, our job is to try to learn
parameters w and b, so that becomes a good estimate of the chance of
y being equal to 1.
z

Logistic Regression Cost
Function
datahacker.rs

 To train parameters W and b of your logistic regression model you need to
define a cost function.
 Given a training set of m training examples, we want to find parameters W
and b so that ŷ is as close to y (ground truth).
 We will use (I) superscript to index different training examples.
 We use loss (error) function (L) to measure how well out algorithm is doing.
Loss function is applied only to single training sample.
007 – Logistic Regression Cost
Function

 Squared Error:
 We don't use this one because it makes gradient descent not work well so in logistic
regression optimization problem becomes non-convex.
Function
3D plot of non-convex
gradient descent
Global minimum

Function
 We will use following loss function:
 It will give us convex optimization problem and it is much easier to otimize.
 Y and ŷ are sigmoid function so they can never bi bigger than 1 or less then 0
 If y = 1:
 L ŷ, = - log ŷ
 Want log ŷ large, want ŷ large (as close to 1 as possible)
 If y = 0:
 L ŷ, = - log (1 - ŷ
 Want log (1 - ŷ large, want ŷ small (as close to 0 as possible)

Function
 Cost function (how well are you doing on the entire training set):
 J(W,b) =
 J is average of sum of loss functions of all parameters.
 Cost function is cost of all parameters.
 We will try to find W and b that minimize the overall cost function J.

Gradient Descent
datahacker.rs

008 – Gradient Descent
 Gradient Descent is a process
of trying to minimize the
error function.
 Gradient descent algorithm is
been used to train and also
learn the parameters W on
your training set.
 In order to learn the set of
parameters W and B it seems
natural that we want to find
W and B that make the cost
function J (W, b) as small.

 Gradient Descent starts at an initial point and began taking steps in the
steepest downhill direction.
 After a si gle step, ou ight e d up there e ause it s tr i g to take a step
downhill in the direction of steepest descent or quickly down low as possible.
 So after one and two iterations of gradient descent, you might end up with
the third iteration and so on, until eventually, hopefully you converge to this
global optimum or get to something close to the global optimum.
How does Gradient Descent Work

 The learning rate
controls how big a step
we take on each
iteration of gradient
descent
 Derivative is basically
the update of the
change you want to
make to the
parameters W
 Note : i our ode d
will be used to
represent this derivative
term.
 Derivative is the slope of a function at the point.
 When derivative is negative.
 When derivative is positive.

 Here the deri ati e is positi e, W gets updated as W i us a lear i g rate ti es
the derivative. We know the derivative is positive, so we end up subtracting from W and
taking a step to the left. Here, gradient descent would make your algorithm slowly
decrease the parameter, if you have started off with this large value of W.
 Here the derivative is negative, the gradient descent update would subtract alpha times
a negative number and so we end up slowly increasing W and end up you're making W
bigger and bigger with successive iterations of gradient descent
 So that hopefully whether you initialize on the left or on the right, gradient descent
would move you towards this global minimum.
Explanation for the graph.

These equations are equivalent.
Partial Derivatives -
RULE : If J is a fu tio
of two variables, instead
of usi g the lo er ase d ,
we use the partial
derivative symbol.

We can see a picture of the
gradient descent algorithm in
action, in a 3D visualization, were
it is taking baby steps in order
a d hopefull to rea h it s glo al
optimum.
Starting point
Stopping point
Visualization in 3D

 By using the function gradient descent we have a general direction in
which to go to reduce the error
 Convex functions have a single global optimum
 Non-Convex functions have different local optima.
 Optimization algorithm tries to alter model parameters “ " and get a
smaller error
 Lots of calculation is based on Linear Algebra
 That means we can write lots of things in a matrix notation
 This allows us to parallelize, it quite a lot on a GPU

009 - Derivatives
 We should open up the box and peer a little bit further into the details of calculus but really all you need is an intuitive understanding of this in order to
build and successfully apply Deep Learning algorithms.
 The term derivative basically means slope of a function.
 Here the function f(a) = 3a is plotted so it's just a straight line to gain intuition about derivatives.
 Slope is a friendlier way to describe the concept of derivative, so whenever you hear derivative just think slope of the function and more formally the
slope is defined as the height divided by the width of this little triangle that we have in blue.
2
6
2.001
6.003
width=0.001
height = 0.003
slope (derivative) of f(a) for a = 2 and a = 2.001 is 3
slope (derivative) of f(a) for a = 5 and a = 5.001 is also 3
Whenever a is nudged by 0.1
or . or . … , slope will
always be 3 for function f(a) = 3a

 It is explained for derivatives what happens that we nudge the variable a by 0.001.
 If you want a formal mathematical definition of the derivatives, derivatives are defined with an even smaller value
of how much you nudge a to the right so it is not 0.001, it is not 0.00001 and so on one, it's sort of even
smallerthan that.
 The formal definition of derivative says what have you nudge a to the right by an infinitely tiny tiny amount for
f(a), a go up three times as much as whatever was the tiny amount.
 One property of the derivative is that no matter where you take the slope of this function, it is equal to 3 whether
a = 2 or a = 5, the slope of this function is equal to 3.
 This function has the same slope everywhere and one way to see that is that wherever you draw little triangle
like that blue in the previous slide, the height divided by the width always has a ratio of three to one.
009 - Derivatives

More Derivatives Example
datahacker.rs

Intuition about Derivatives
010 – More Derivatives Example
2 2.001
4.004
4
0.004
0.001
 We have plotted the function f(a) = a², taking a look
at the point a=2, a² or f(a) = 4.
 If we move a slightly to the right, a=2.001, f(a) which
is a² is going to be approximately 4.004.
 What this means is when a=2?
 What we're saying is that if a=2, then f(a) = 4 (here
the x and y axis are not drawn to scale).
 If we move a to 2.001, then f(a) becomes roughly
4.004.
 So if we draw a little triangle, this means is that if we
move it to the right by 0.001, f(a) goes up four times
as much by 0.004.

2 2.001
4.004
4
0.004
0.001
 In the language of calculus, we say that a
slope that is the derivative of f(a) at a=2 is 4
 Calculus notation:
 We say that d/da of f(a) = 4 when a=2.
 Now one thing about this function f(a) = a² is
that the slope is different for different values
of a.

2 2.001
4.004
4
0.004
0.001
 If a=5, then a²=25, so that's f(a).
 If we move a to the right again, it's tiny little
change to a, now a=5.001, then f(a) will be
approximately 25.010.
 So what we see is that by moving a up by 0.001,
f(a) goes up ten times as much 0.010.
 We have that:
 When a=5 because f(a) goes up ten times as
much as a does when we make a tiny little nudge
to a.

Computation Graph
datahacker.rs

 The computations of a neural network are organized in terms of
 a forward pass or a forward propagation step, in which we compute the
output of the neural network,
 followed by a backward pass or back propagation step, which we use to
compute gradients or compute derivatives.
 The computation graph explains why it is organized this way.
 In order to illustrate the computation graph, let's use a simpler
example than logistic regression or a full blown neural network.
011 – Computation Graph

 Let's say that we're trying to compute a function J, which is a function of
three variables a, b, and c and let's say that function J is 3(a+bc).
 Computing this function actually has three distinct steps:
1. Compute bc and store it in the variable u, so u=bc
2. Compute V=a u,
3. Output J is 3V.
J(a,b,c)=3(a+bc)
u=bc
V=a+u
J=3V
a
b
c
u = bc
V = a+u J = 3V

 So, the computation graph comes in handy when there is some
distinguished or some special output variable, such as J in this case, that
you want to optimize.
 And in the case of a logistic regression, J is of course the cost function
that we're trying to minimize.
 And what we're seeing in this example is that, through a left-to-right
pass, you can compute the value of J.

Derivatives with a
Computation Graph
datahacker.rs

 We have learned how we can use a computation graph to compute the
function J.
 How we can use a computation graph to figure out derivative calculations
of the function J.
 Using a computation graph we want to compute the derivative of J with
respect to v.
012 – Derivatives with a Computation Graph
a
b
c
u = bc
V = a+u J = 3V
11 33
a=5
b=3 u=6
c=2

 So what is that?
 If we were take this value of v and
change it a little bit, how would the
value of J change?
 Analogous example a
b
c
u = bc
V = a+u J = 3V
11 33
a=5
b=3 u=6
c=2
Forward propagation
Back propagation
Calculating dJ/dv is a one
step of back propagation.

 What is dJ/da?
 That is, if we increase a from 5 to 5.001, v will increase to 11.001 and J will
increase to 13.003
 So the increase to J is three times the increase to a so that means this
derivative is equal to 3.

 One way to break this down is to say that if you change a then that would
change v and through changing v that would change J.
 By increasing a, how much is v increased? This is determined by dv/da.
 Then, the change in v will cause the value of J to also increase. This is called
a chain rule in calculus:

 In this course we will use the following notation:
 So, in our code will use:

 What is the derivative dJ/du?
 A question when working with derivatives is:
How much do we need to change b in order
to maximize/minimize J? What is the slope
when we change b?
 When we change b, we will change u, and u
will affect J.

 What is the derivative dJ/dc?

Logistic Regression
Derivatives
datahacker.rs

013 - Logistic Regression Derivatives
Logistic Regression Recap
 Z - Logistic Regression Formula
 Y predictions
 Loss Function where a is
the output of logistic
regression and y is the
ground truth label

Explanation for the diagram
 In this example we have only two features x1 and x2. So in order to compute Z,
we'll need to input w1, w2 and B in addition to the feature values x1 and x2 :
 Then we compute y-hat or a equals Sigma of Z : or
 Finally we compute our loss function :

 In logistic regression all what we want to do is to modify the parameters W
and B in order to reduce this loss.
 First we have to compute the loss on a single training using the forward
propagation steps.
 Then we go backwards to compute the derivatives.

How do we compute the derivatives.
 Compute the derivative of and in code we would denote this as
Note he riti g ode I assig ed to da
 Ha i g o puted da , e a go a k ards to o pute dz
Since we know that :

 The final step in back propagation is to go back to compute how much we need to
change w and b
 so if you want to do gradient descent with respect to just this one example, we would do
the following by performing this updates.

Gradient Descent on m
Training Examples
datahacker.rs

014 – Gradient Descent on m Training
Examples
 The derivative respect to w1 of the overall cost function is also going to be the
average of derivatives respect to w1 of the individual loss terms.
 Previously we have already shown how to compute this term for a single training
sample:
 We need to compute this derivative on the previous training example and average
them and we will get the overall gradient that we can use to implement gradient
descent.

Examples
 Let's wrap this up into a concrete algorithm and what you should implement to get
logistic regression with gradient descent working. We do this calculation assuming
that we have two features so that n = 2, otherwise we do this for dw1, dw2, dw2...
For i = 1 to m
n

Examples
 After we leave for loop we need to divede all by m because we are computing
averages:
 With all of these calculations, we computed derivative of the cost function J with
respect to parameters w1, w2, and b.

Examples
 We're using dw1 and dw2 and db as accumulators.
 We use them to sum the entire training set.
 After all this calculations to implement one step of gradient descent we implement

Examples
 It turns out there are two weaknesses with calculation as we've implemented in
here.
 To implement logistic regression this way we need to write two for loops (loop over
m training samples and loop over all the features).
 When we are implementing deep learning algorithms we find that explicit for loops
make our algorithms run less efficiency so if we want to scale to bigger data sets we
must avoid them.
 For that we use vectorization.

015 - Vectorization
 Vectorization is a style of computer programming where operations are applied
to whole arrays instead of individual elements
 Basically the art of getting rid of explicit for loops in your code.
 Deep learning algorithms tend to shine when training on large datasets, so it's
important that your code runs quickly. Otherwise, your code might take a long
time to get your result.
 Vectorization can significantly speed up your code.
 The rule of thumb to remember is whenever possible avoid using explicit for
loops in your code.
 In the next slides, I would be doing a comparison in time using a vectorized and
non-vectorized implementation to compute the dot products of two arrays.

015 - Vectorization
Vectorized Implementation.

015 - Vectorization
Non-Vectorized Implementation.

015 - Vectorization
 Our results clarifies that having a vectorized implementation in your code, does
actually make your code run faster.
 The vectorized Implementation took 597 nanoseconds ( Better )
 While, the non-vectorized implementation took 407 milliseconds.
 NumPy is the fundamental package for scientific computing in Python.

More Vectorization Examples
datahacker.rs

 When we are programming neural networks or logistic regression we
should avoid explicit for loops.
 It isn't always possible but when we can we should use built-in functions or
find some other ways to compute.
 Some examples:
 Non vectorized: Vectorized:
U=np.dot(A,V)
016 – More Vectorization Examples

 Let's apply the exponential operation on every element of matrix/vector.
 Non vectorized:
 Python library numpy has many functions that help us with vectorization.
 Vectorized:
Import numpy as np
U = np.exp(v)

 Some other numpy vectorization functions:
 Np.log
 Np.abs
 Np.maximum
 V**2
 1/v

 Let's apply this on our logistic regression gradient descent to remove our
for loops.
 We still have one for loop but code is running much faster.

Vectorizing Logistic
Regression
datahacker.rs

017 - Vectorizing Logistic Regression
 We are going to learn how to vectorize our implementation of logistic regression
 We want to process an entire training set that has implemented a single iteration
of gradient descent with respect to the entire training set without using even a
single explicit for loop.
 First we need to compute the forward propagation step of logistic regression, by
computing this below :
 First we need to compute Z, then compute also the activation function.
 For M training examples, you have to carry out the forward propagation step that
is to compute these predictions on all M training example.

 First we stack together in different columns our training inputs and assign them to
the variable which is an Nx by M dimensional matrix.
 (w transpose) would be a row vector.
 b is a 1 by M matrix or an M dimensional row vector.
 Next we can compute our Z as a 1 by M matrix
 In python :

 Next we want to find a way to compute A all at the same time.
 Formula :
 In python :

Vectorizing Logistic Regression's
Gradient Computation
datahacker.rs

 We are going to learn how to compute all samples at a same time.
 We need to define new variable dZ which will contain all dz values stacked
horizontally in 1 by m matrix.
 In the previous example, we have gotten rid of one for loop but the second
one (for loop over m training samples) still remained.
018 – Vectorizing Logistic
Regression's Gradient Computation

 Now we will vectorize all. Starting with vectorization of db and dw.
 In python:

 Let's pull it all together to actually implement logistic regression.

 With this, we implemented single iteration of logistic regression.
 If we want to implement logistic regression over a number of iteration we
need for loop.
 We can't get rid of that for loop.

Notes from Coursera Deep Learning courses by Andrew Ng

More Related Content

What's hot

Similar to Notes from Coursera Deep Learning courses by Andrew Ng

More from dataHacker. rs

Recently uploaded

Notes from Coursera Deep Learning courses by Andrew Ng