lasso regression via fminsearch not shrinking weights

Hi all,

I'm trying to apply the lasso regression equation in matlab to give output weights so that once i'm sure this is working i can try applying different penalty terms from the l1 norm for my own research. When i try applying a range of lambda, the tuning parameter, the output weights should go to zero over time, meaning i should have no zeros, then 1, then 2 etc. until all are zero. However instead all values are remaining as non zeros and eventually all change to zeros at once, i tried applying the lambda values calculated via the lasso command, as via the lasso command the weighs shrank as expected, but for these lambda values my fminsearch code did the same as before, no zeros then all zeros.

I'm using the lasso for an extreme learning machine so the H value is my input from my randomised nodes, the fminsearch works fine without the l1 norm constraint, but once added it does not shrink the weights as expected. My code of this section can be seen below.

   i = 1;

for TuningParameter = 0:0.0001:0.1

   f = @(OutputWeight,TuningParameter,Target)((sqrt(mse(Target-(H*OutputWeight))))+(TuningParameter*(norm(OutputWeight,1))));

   Target = YNoiseyTrain;

   fun = @(OutputWeight)f(OutputWeight,TuningParameter,Target);

   OutputWeight0 = (zeros([1 HiddenNodesNumber]))';

   OutputWeight = fminsearch(fun,OutputWeight0);

   S(:,i) = OutputWeight;

   Y(:,i)= H*OutputWeight;

   s = i+1;
   i = s;
  end

I've also uploaded 2 files, one showing how my output matrix should look with shrinkage as the tuning parameter increases and the other showing how the fminsearch output currently looks.

Any help would be great, thanks in advance.

Answer 1

[Matt J]

You haven't provided enough info to run your code or to know the sizes of the input data, so I will venture two guesses,

1. You are applying fminsearch to a function called fun2 which is defined nowhere in the code that you've shown.
2. You are trying to minimize over too many parameters. FMINSEARCH is only guaranteed theoretically to converge for one unknown parameter, and empirically tends to work for up to about 6 parameters.

[Jake Rainey]

Sorry the fun2 was a typo on this post, it's just fun which i've corrected and the FMINSEARCH should be focusing on changing the parameter Outputweight, i've specified all other parameters, some before this bit of code i've posted.

It runs fine with no errors it's just the Outputweight matrix given by the function should have 5 rows which shrink at different speeds to 0, so the number of 0s should increase over time, instead the function sets all to 0 at once.

Thanks for the help.