This behavior occurs when the QPSUB function (a private routine in the Optimization Toolbox) becomes stuck in an infinite loop. This behavior is called 'zigzagging' and affects QPSUB (hence QUADPROG and FMINCON: medium-scale).
When this occurs, set the "MaxSQPIter" property (if you're running FMINCON) or the "MaxIter" property (if you are running QPSUB).
For example, a possible value could be:
10*max(numberOfVariables, numberOfInequalities + numberOfBounds);
This can be performed as follows:
M = 10*max(numberOfVariables, numberOfInequalities + numberOfBounds);
options = optimset('MaxSQPIter', M);
This should stop MATLAB from hanging and rather, in the QUDPROG function, will terminate the process with a warning that the maximum number of iterations was reached. At this point, you should, check that the answer obtained is accurate for your purpose.
In general, setting the maximum number of iterations to an finite number in QPSUB is not uncommon in optimization. This should not have a negative impact in the overall minimization. The QPSUB function computes an intermediate step, and there is a check in the main optimization algorithm that will reject this step if it is inadequate. However, ensure that you perform checks on a case-by-case basis.
