Linear regression, L2 regularization

charbsj
Posts: 7
Joined: Sun Jan 20, 2019 1:55 pm

Re: Linear regression, L2 regularization

Post by charbsj » Wed Feb 20, 2019 2:02 am

I have 3 cases.
Case 1: no outliers
Case 2: outliers on the high side
Case 3: outliers on the low side

What I mean by “it does not work” is that all the L2 regularization is doing (in my code) is “penalize” high weights. This results in the weight being lower than they would be without L2 regularization. This is ok for Case 1, it’s great for case 2, it is counter productive for case 3

If you click on the link, there should be the option to go to code or go to file. Then, there is py file.
Can you find it?

Toward the middle there is 15 lines of comment explaining the 15 different test cases.

Thank you for your willingness to help

Felipe_Brazil
Posts: 2
Joined: Fri Dec 21, 2018 6:23 pm

Re: Linear regression, L2 regularization

Post by Felipe_Brazil » Thu Feb 21, 2019 11:00 pm

I was also getting this conclusion, but i'am not sure we share the same problem.

The L2 regularization results depends in the sum of two parameters: sum of weights and r-squared value (minimal average distance from Y_hat line to all original points) .

The thing is, sometimes when we have situation A and situation B the sum of weights are equal, but the square error is not.When it happend I got the same impression at first, when positive points were closer to my "Y_hat line" and negative points were not, focusing only in the sum of weights i didn't notice the other parameter (square error) varying. I couldn't confirm that hypothesis by reading your code.
Perhaps you could compare the "r-squared" and the "sum of the square of the weights" values for both situations?

Other possibility may be that the distances (from your outliers to the curve) don't have gausian distribution, but normal instead (If I didn't miss understand the lazyprogrammer class the distribution could be problem).
charbsj wrote:
Wed Feb 20, 2019 2:02 am

If you click on the link, there should be the option to go to code or go to file. Then, there is py file.
Can you find it?
Yes, I can :) .
But my beginner Python skills couldn't help me clearly understand it as i wanted. :lol:
I get a bit confused with the concept of "classes". :|

Wish I could help you in some way.

Post Reply

Return to “Deep Learning Prerequisites: Linear Regression in Python”