When the algorithm of Levenberg-Marquardt is preferred when doing curve fitting?

I would like to do a curve fitting, and there're 2 algorithms in the curve fitting toolbox. The trust-region is default,and it can provide me with satisfying fitting as shown below. However, I am now sure when the other algorithm (Levenberg-Marquardt) is preferred, though I've checked the help page (https://uk.mathworks.com/help/optim/ug/equation-solving-algorithms.html) of those 2 algorithm, where only the basic principles are explained without the applicable conditions of them. Could anyone please explain the differeces and the applicable conditions of the 2 algrithms?

Answer 1

[Bruno Luong]

Trust region is more robust if you have strong non-linearity. This effect is "amplified" depends also how far the starting point from the true solution.

The downside is it project the "Hessian" on a small subspace (2nd dimension), so it will not converge rapidly if the function is convex but with large difference of the amplitude of the curvatures. But it can deal with with local negative curvatures.

Levenberg-Marquardt requires to evaluate the Jacobian, which can only effectively computed in small/middle scaled problem. As I said, it's approximate the Hessian with J'*J, so it's more accurate for problem with medium non-linearity.

In short: Trust region more robust, used for large scale, strong non-linearity. (typo EDIT)

Levenberg-Marquardt , less robust, used for medium scale, medium linearity, or the first guess is well estimated.

[Shuangfeng Jiang]

Thanks for your detailed anwer!

1. I am not sure I uderstand 'large scale' and 'medium scale' corectly: Does large scale mean the initial point/ guess may be far from the optimal? Or does small/medium scale mean that we can adopt it when our initial guess is close to the optimal? I've also checked the corrsponding help page where the princeples of those are explained, but the applicable conditions are not given.
2. At the begining of your aswer: you said"Trust region is more robust if you have strong non-linearity", while in the sumary: "Trust region more robust, used for large scale, strong linearity", then I may assume you are correct at the beginning ?

[Bruno Luong]

In optimization, the "large-scale" mean the numbers of parameters (under optimization) is large. For constrained optimization case, it includes also the number of (active) constrained.

Because if the gradient is not provided, the optimization must run the model many times (the number of parameters) to compute the model jacobian needed for Levenberg-Marquardt, and also optimizarion has more difficulty to deal with non-linearity, etc...

You are right about the typo on non-linearity.

[John D'Errico]

Yes. Large scale means many variables. If you are doing curve fitting, it is hardly ever true that you would enter into the domain of a truly "large scale" problem. 3 or 4 or 6 variables is not even medium scale.

If you have a poorly chosen initial point, this can create problems, since with a nonlinear function, you can put the optimizer into nasty places that it may not easily escape. And that can create problems with "large residuals", another key phrase to remember. Large residual nonlinear least squares problems are classically bad. They cause issues because a single bad point can jerk the optimizer around. It willl attempt to minimize that residual while virtually ignoring the rest, since it can get the most bang for the buck on one data point.

The best solution is to have good data, and to supply good starting values. Then it won't matter what algorithm you use.

[Shuangfeng Jiang]

Thanks a lot for your anwer!

Just final reminding that you may need to modify another typo in the summary of Levenberg-Marquardt which is preferred with medium non-linearity.

[Bruno Luong]

well not necessary "medium linearity ~= medium non-linearity", but all these notions of "medium", "strong" "large scale" is emperical in maths I must confest.