Multiple Linear Regression - How do we know its the local minimum of the error

Post Reply
guyo13
Posts: 2
Joined: Tue May 16, 2023 8:12 pm

Multiple Linear Regression - How do we know its the local minimum of the error

Post by guyo13 »

Hi,

I took the lecture about the multi dimensional linear model.

I understand that we are trying to minimize the Error function we chose for the model.

What I don't understand are some of the assumptions:
1) That solving the linear system we got by taking the partial derivatives and comparing them to 0 - will yield the weights vector that will specifically minimize the error? How can we assume that the error function has its local minima there (eg. why not local maximum)?

2) How can we assume that X.T.dot(X) is an invertible matrice?

Thanks!
lazyprogrammer
Site Admin
Posts: 127
Joined: Sat Jul 28, 2018 3:46 am

Re: Multiple Linear Regression - How do we know its the local minimum of the error

Post by lazyprogrammer »

Thanks for your inquiry.

1) One way to see is to plot it. Another way to see is to use the methods taught in my latest Calculus course. https://deeplearningcourses.com/c/calculus-data-science

(However it also requires linear algebra knowledge, but I haven't released that course yet)

2) It's not always, and in fact this is discussed in the course!

More prerequisite details will be included in the coming linear algebra course.

However, note that you'll get an error if the matrix is not invertible. Therefore, checking is easy.
guyo13
Posts: 2
Joined: Tue May 16, 2023 8:12 pm

Re: Multiple Linear Regression - How do we know its the local minimum of the error

Post by guyo13 »

Thank you for the amazingly quick response.

Regarding point (2), got it. So at first glance that means that there is a dependence between the training data and whether we can even solve the regression. I will explore this dependence as HW.

Regarding point (1), my curiosity wasn't satisfied - and I'm only after Calculus 2 so only introductory knowledge in 2-variate differential calculus.
I chatted with ChatGPT and there seems to be a "second derivative test" that examines a "Hessian matrix" of the error function and its eigenvalues.

I feel I can explore these methods on my own but I'm curious to what degree do you cover this material in your calculus course?
I'm a CS undergraduate student (and full time software engineer) so I want to make a good time investment if I'm to take it :D.

Thanks!
lazyprogrammer
Site Admin
Posts: 127
Joined: Sat Jul 28, 2018 3:46 am

Re: Multiple Linear Regression - How do we know its the local minimum of the error

Post by lazyprogrammer »

2) has more to do about the "rank" of the data matrix (X), which is related to the number of linearly independent columns (i.e. features).

1) the calculus course makes the same statement (because that's the only statement which needs to be made) but obviously in more detail, e.g. with visual intuition.

In order to explore it more you would need to know what eigenvalues are, and what is meant by "positive definite" and "negative definite" which will be taught in the linear algebra course (Hessian is already covered in the calculus course).
Post Reply

Return to “Deep Learning Prerequisites: Linear Regression in Python”