In my previous post , we were discussing about finding the derivative of a linear function. The derivative of the function (y = 2 * x) was 2 and it remained constant for what values of x and y you choose and substitute in the formula for finding dy/dx .But there are some functions whose derivative will not stay constant but depend on the point where you are evaluating.To understand the above statement , we will need to understand some stuffs from trigonometry. Let us understand these by pictures.
In the above picture , a right triangle whose hypotenuse of length “1” is considered. By definition , tan(theta) is defined as the ratio of the Opposite side to its Adjacent side(Opposite side / Adjacent side).When we compute tan(theta) it comes out to be ratio of y to the x or ( y2 – y1)/(x2 – x1 ). If we recall in our previous post, it’s nothing but the derivative of a function. Tan(theta) is also defined as the slope of a line or the angle which the line makes with the x-axis.If the theta value is increased then the line would move towards y-axis and hence the derivative of the function increases(Why?).There are lot things to digest,so pause yourselves a while and ponder over it.So ,till now we discussed about finding the derivative of a linear function.Lets take a step further and talk about non-linear function.Before moving on, we must first understand what is a non-linear function?.A non-linear function is a function whose points when plotted and joined, does not form a straight line(may be a curve..).Lets call our friend Mr.Graph.
The above function is not linear , because when the points are joined it forms a curve.Try finding the derivative of a function yourselves.If you try to find you would not get the correct answer.Why?.Lets understand it visually.
If you look into the above image , I used the same method for calculating the derivative of the function as I did in the previous example.If you watch closely , you would see the hypotenuse of the triangle is quite away from the curve.What does this tell??.If the function was linear then the hypotenuse of the triangle is nothing but the function’s line.To understand it simpler , we would consider a small example.
X 0 1 2 3
Y 0 1 4 9 ( Y = X * X -> Non-linear function )
X 0 1 2 3
Y 0 2 4 6 ( Y = 2 * X -> Linear Function )
In the linear function , If I change x from 1 to 2 then correspondingly the y value changes from 2 to 4 . So the ratio is (4 – 2)/(2 – 1) which is 2 .If you consider any other x and y, you would get the same ratio. But in the first function( Y = X * X ) , If I change x from 1 to 2 the y value changes from 1 to 4 and similarly if i change x from 2 to 3 the y value changes from 4 to 9.If you compute the above ratios , you would find that they are not the same. So ,the derivative of the function does not stay constant and we need to come up with a new strategy to find it.Before moving onto finding the derivative, we would summarize ourselves till now.Derivative of a function is nothing but , the ratio of change in output to the input or measuring how much the output changes if i change the input!.That’s all is about the derivative of a function.
To gain more intuition about derivative of a function ,we will now consider an real world problem where the derivative of a function is used.Let us plot the distance covered by a car in the X-axis and time it took in Y-axis.Let’s call our friend Mr.Graph
Average Speed is calculated by dividing the total distance travelled by the time taken.It’s comes out to be the derivative of the function.So derivative of the function tells us the rate of change of function or how much the output changes when the input changes.In the above example, the speed remains constant(2) and doesn’t change.But this will not be always the case.We can’t drive a car at constant speed all the time.We vary our speed at some point and sometimes we stay constant.In this case,we cannot calculate the average speed, because our speeds are different at different points.Hence,the derivative will not stay constant.We previously discussed in this post about non-linear function and its derivative.We noticed that the ratios were different in each case.So to overcome this , we are going to find the speed(rate of change) at a point and not between two points.Since speed is changing at each point , what if we found a formula for finding speed at each point rather than between any two points?Sounds good??..Let us look at the graph of y = x * x again.
In the above picture when we change our x from 1 to 2 the y value changes from 1 to 4.But we also observed that the hypotenuse didn’t fall on the curve, rather it is quite away from the curve.It’s because the speed is not constant and changing at each points and hence we cant draw a line and connect those points.We already know how to compute the derivative of a linear function.If we make this function to be linear, then our life would be easier.How to make?.Its easy if we closely watch the triangle.If we make the dx smaller then the line would move towards the curve and hence it would approximate the curve.Then if the line coincides with the curve, then it would form a straight line(linear function) and we can easily find its derivative.So how small the dx should be to approximate the curve?.What if the dx were 0 ? It can’t be.Why? Because rate of change is defined as dy/dx and hence if dx is 0 then dy/dx is undefined.So we need a number which is closer to 0 but not zero.How to express the former statement mathematically. Limitssssssssssssssssssss………………….
Does it make sense?. If we make dx as close to zero , then the hypotenuse would approximate the curve better and hence it is linear at that point.So, we know how to find the derivative for that case . f(x0 + del x) – f(x0) = y2 – y1 . Why ? Lets find it in the graph.
I think the image is clear enough.If we make dx closer to 0 then the tangent line would come closer to the curve and changes to linear at that point.So to calculate the derivative at that point we would substitute x0,x0+dx,f(x0),f(x0+dx) in the above limit equation.
Woww..we found the derivative of the function at a point x.When we compare this with that of the linear equation, we find that the derivative of this function(y = x^2) is dependent on x and it’s not constant like in y= m* x(where m may be 0,1,2…).So for a exercise if we find what is the rate of change at point 2 , it comes out to be 2 * 2 = 4(f ‘(x) = 2x) and at point 3 it is 6 (f ‘ (x) = 2x) .So the speed is changing from point 2 to point 3 and it is not constant.Thus, it is a non-linear function.With this enough details, we can talk about gradient descent algorithm using backpropagation in the next post.
Harish's Blog 