Hi, any help would be appreciated -- MLE never errors but simply becomes stuck at some iteration when using a custom PDF. Furthermore, this custom PDF is a modification of a previously working custom PDF, and I'm not quite sure what it is that could be causing the problem.
In more detail:
I'm using mle with a custom PDF, with fmincon. The PDF is complicated, but in the basic version essentially comes down to a mixture of several truncated normal distributions. This works fine.
I've recently modified the PDF to use a doubly stochastic distribution, ie, instead of estimating the standard deviation of a truncated normal, I am using mle to estimate the shape and scale parameters of a gamma distribution from which the standard deviation of the truncated normal is then drawn. A single call to the PDF function results in the drawing of hundreds (or thousands) of SDs from a gamma pdf with the tested shape and scale parameters. These SDs are then used as input to the truncated normal, and the probabilities of these truncated normals are then averaged to produce an estimate of a mixture of truncated normals over the entire gamma distribution of SDs. As both a reference for why I might want to do this, and some proof that this can be done in principle, see this 2012 article in Proceedings of the National Academy of Science by van den Berg and colleagues. I'm calling mle using 'display', 'iter' and it appears to work -- I get output like:
Norm of First-order
Iteration f(x) step optimality CG-iterations
0 511.215 1.5e+04
1 511.215 0.254905 1.5e+04 0
2 511.215 0.0266765 1.5e+04 0
3 511.215 0.00666913 1.5e+04 0
4 508.887 0.00166728 1.96e+04 0
5 508.887 0.00041682 1.96e+04 0
6 508.643 0.000104205 4.68e+04 0
However, that's it! Sometimes I can get a complete run through and get likelihoods from the MLE. A single iteration of mle (as measured by simply watching the displayed iterations come up) takes anywhere from << 1 second to a couple seconds. But I'm using this mle on many sets of data (separate conditions from separate research participants), and eventually, usually sooner rather than later, I get to one of these iterations that simply goes nowhere. On my own computer, I've let this run for many hours with no progress. On our computing cluster, I currently have four separate attempts at this (with minor tweaks to things like the error tolerance and the number of draws from the gamma) running, all of them for ~ 3 days, all hung in this same way. I've also tried multiple algorithms (interior-point and sqp, specifically) for the mle.
Again, I'm at a bit of a loss because there is no error, there is no warning... nothing for me to debug, just an absence of progress.
Does anyone have any suggestions for either the cause of this issue, or how to go about debugging such a no-error issue? If it were my code that was hanging, I'd worry about an infinite loop somewhere, but this seems to be in the mle function or something it calls, and I'm quite sure that there's no infinite loop in my pdf.
Thanks in advance for any advice, Ben
