How to extrapolate to outlying unknown datapoints in an 2D matrix

Hello,

I have an experimental dataset on a fixed grid that is incomplete due the upper limit of the measuring equipment having been reached. For example:

[8 8.5 9 9.5 10 10 10 10 10 10;
7 7.5 8 8.5 9 9.5 10 10 10 10;
6 6.5 7 7.5 8 8.5 9 9.5 10 10;
5 5.5 6 6.5 7 7.5 8 8.5 9 9.5]

where 10 is the limit of the equipment (Clearly my example is trivial but my data isn't linear!)

I have tried replacing all the unknowns with NaNs, then using interp2 but I couldn't think around a way to handle the NaNs in the array and essentially fit to the same grid. I use Inpaint_NaNs for interpolation, and gave this a go for this problem, but the results visually were incorrect.

Anyone have a solution please?

Many thanks, Peter

[Star Strider]

I don’t understand the data you posted. Can you post your code and data you’re working with?

As a general rule, extrapolation isn’t a good idea. You’re stuck with what you can measure.

Answer 1

[John D'Errico]

I'll bite. Assuming you replaced the 10s with nans,

A = [8 8.5 9 9.5 10 10 10 10 10 10;
     7 7.5 8 8.5 9 9.5 10 10 10 10;
     6 6.5 7 7.5 8 8.5 9 9.5 10 10;
     5 5.5 6 6.5 7 7.5 8 8.5 9 9.5];

A(A == 10) = NaN;

Ahat = inpaint_nans(A)
Ahat =
  8   8.5   9   9.5   10   10.5   11   11.5   12   12.5
  7   7.5   8   8.5    9    9.5   10   10.5   11   11.5
  6   6.5   7   7.5    8    8.5    9    9.5   10   10.5
  5   5.5   6   6.5    7    7.5    8    8.5    9    9.5

Personally, I think inpaint_nans did a hell of a good job here, even if I do seem immodest by so saying. Perhaps you can say why it is not to your liking?

(I can only assume that your surface is a bit more nasty, and/or the extrapolation goes out much further. Regardless, he who tries to predict too far into the future had better get a good crystal ball. The same goes for large amounts of extrapolation.)

Edit: Based on Peter's actual data, I can go a bit further...

First, the original surface. The arrow points to what may possibly have been a subtle problem with the inpainted surface.

Next, the surface as produced by the default method.

And the surface that method 3 generates.

See that subtle variations in the corners of the data will be sufficient to make the extrapolant wobble the farther out you go. This may happen more with method 3 than with the default, because method 3 uses a higher order method.

Personally, I like the default prediction. I'm still unsure why you don't like the prediction here. It seems consistent with the initial surface fragment. No matter what, extrapolation is a process fraught with danger.

What did you expect to happen there?

[John D'Errico]

Do let me know if there is something we might achieve to fix your problem, but I need to understand what you are looking for here. The problem with extrapolation is it is a lot like the butterfly effect. Tiny changes on an edge or corner of a "hole" in the array will result in larger perturbations away from there. Extrapolation falls into the general category of problems where the solution tends to magnify such tiny perturbations.

[Peter]

Hi John,

I think what I expected was the smooth completion of the two rows with NaNs in them, following the trend of those below. Hence, the problem with method 0, to me, is that the final row ends up 'lumpy', as if the reduced relative increase of the data known in that row is ignored in that extrapolated.

Actually to my eye, method 3 is somewhat better, with the exception of the corner point(s) - this is closer to my imagined version.

Anyway thank you for your consdieration of this, what I take away from this is that "No matter what, extrapolation is a process fraught with danger." Wise words indeed.

Peter

[John D'Errico]

Ok. Yes, that final row is a bit bumpy, no matter what method is used. But it really is a butterfly effect, where tiny variations along the edges will impact the extrapolated values, and be greatly magnified in an unpredictable fashion. The methods involved are designed to be smooth, in that they use partial differential equations that inherently involve smoothing operators. (The main methods I've chosen use elliptic operators.) Thus other methods of inpainting may be worse in this respect. For example schemes that employ more terms from the Navier-Stokes equaions will often include hyperbolic terms. This is nice in that it allows wave-like action, however it will also be less smooth in extrapolation.

Answer 2

[Andrei Bobrov]

    A = [0.000577900000000000,0.000585500000000000,0.000595090000000000,0.000603910000000000,0.000614910000000000,0.000623320000000000,0.000634190000000000,0.000648350000000000,0.000664800000000000,0.000676690000000000,0.000690250000000000,0.000711140000000000,0.000728960000000000,0.000750450000000000,0.000772710000000000,0.000796110000000000,0.000824150000000000,0.000849310000000000,0.000879470000000000,0.000907450000000000,0.000946520000000000,0.000976680000000000,0.00102350000000000,0.00108050000000000,0.00113000000000000,0.00124250000000000,0.00134750000000000,0.00197600000000000;0.0165500000000000,0.0165000000000000,0.0164350000000000,0.0164250000000000,0.0163800000000000,0.0163200000000000,0.0162550000000000,0.0161750000000000,0.0160060000000000,0.0158810000000000,0.0156350000000000,0.0154250000000000,0.0151350000000000,0.0148050000000000,0.0144500000000000,0.0142584109171598,0.0136650000000000,0.0132700000000000,0.0128650000000000,0.0124600000000000,0.0124300000000000,0.0123550000000000,0.0121750000000000,0.0119444821301775,0.0115500000000000,0.0107500000000000,0.00990390000000000,0.0105200000000000;0.0548150000000000,0.0544910000000000,0.0541950000000000,0.0539550000000000,0.0536500000000000,0.0532910000000000,0.0529060000000000,0.0524960000000000,0.0519450000000000,0.0512850000000000,0.0505260000000000,0.0497110000000000,0.0487750000000000,0.0477700000000000,0.0466460000000000,0.0454040000000000,0.0441360000000000,0.0427860000000000,0.0413760000000000,0.0398850000000000,0.0383610000000000,0.0367950000000000,0.0352610000000000,0.0337600000000000,0.0323300000000000,0.0309550000000000,0.0296950000000000,0.0296350000000000;0.0926590000000000,0.0922050000000000,0.0916750000000000,0.0912090000000000,0.0906560000000000,0.0900610000000000,0.0893710000000000,0.0885850000000000,0.0876050000000000,0.0865400000000000,0.0852810000000000,0.0838810000000000,0.0823520000000000,0.0806810000000000,0.0788190000000000,0.0768050000000000,0.0747570000000000,0.0724750000000000,0.0701940000000000,0.0678370000000000,0.0653990000000000,0.0629460000000000,0.0605200000000000,0.0581510000000000,0.0558850000000000,0.0537300000000000,0.0516400000000000,0.0503750000000000;NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,0.0977590000000000,0.0947190000000000,0.0916060000000000,0.0884740000000000,0.0853270000000000,0.0822110000000000,0.0791400000000000,0.0762710000000000,0.0735360000000000,0.0709190000000000,0.0685620000000000;NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,0.0994000000000000,0.0959210000000000,0.0925600000000000,0.0894320000000000,0.0864410000000000,0.0834920000000000;]

[m,n] = size(A);
[yy,xx] = ndgrid(0:m-1,0:n-1);
t = ~isnan(A);
F = scatteredInterpolant(xx(t),yy(t),A(t),'natural','linear');
[X,Y] = ndgrid(0:.1:m-1,0:.1:n-1);

mesh(X,Y,F(X,Y));

Answer 3

[Peter]

Hi John, thanks for you quick response. Yes its a tad nastier... As you say it is most likely the problem of a) asking too much, and b) not having a large enough 'mesh' (as such, this is the best I can do experimentally!). OK, so below is the data, x and y are linear spacing.

[0.000577900000000000,0.000585500000000000,0.000595090000000000,0.000603910000000000,0.000614910000000000,0.000623320000000000,0.000634190000000000,0.000648350000000000,0.000664800000000000,0.000676690000000000,0.000690250000000000,0.000711140000000000,0.000728960000000000,0.000750450000000000,0.000772710000000000,0.000796110000000000,0.000824150000000000,0.000849310000000000,0.000879470000000000,0.000907450000000000,0.000946520000000000,0.000976680000000000,0.00102350000000000,0.00108050000000000,0.00113000000000000,0.00124250000000000,0.00134750000000000,0.00197600000000000;0.0165500000000000,0.0165000000000000,0.0164350000000000,0.0164250000000000,0.0163800000000000,0.0163200000000000,0.0162550000000000,0.0161750000000000,0.0160060000000000,0.0158810000000000,0.0156350000000000,0.0154250000000000,0.0151350000000000,0.0148050000000000,0.0144500000000000,0.0142584109171598,0.0136650000000000,0.0132700000000000,0.0128650000000000,0.0124600000000000,0.0124300000000000,0.0123550000000000,0.0121750000000000,0.0119444821301775,0.0115500000000000,0.0107500000000000,0.00990390000000000,0.0105200000000000;0.0548150000000000,0.0544910000000000,0.0541950000000000,0.0539550000000000,0.0536500000000000,0.0532910000000000,0.0529060000000000,0.0524960000000000,0.0519450000000000,0.0512850000000000,0.0505260000000000,0.0497110000000000,0.0487750000000000,0.0477700000000000,0.0466460000000000,0.0454040000000000,0.0441360000000000,0.0427860000000000,0.0413760000000000,0.0398850000000000,0.0383610000000000,0.0367950000000000,0.0352610000000000,0.0337600000000000,0.0323300000000000,0.0309550000000000,0.0296950000000000,0.0296350000000000;0.0926590000000000,0.0922050000000000,0.0916750000000000,0.0912090000000000,0.0906560000000000,0.0900610000000000,0.0893710000000000,0.0885850000000000,0.0876050000000000,0.0865400000000000,0.0852810000000000,0.0838810000000000,0.0823520000000000,0.0806810000000000,0.0788190000000000,0.0768050000000000,0.0747570000000000,0.0724750000000000,0.0701940000000000,0.0678370000000000,0.0653990000000000,0.0629460000000000,0.0605200000000000,0.0581510000000000,0.0558850000000000,0.0537300000000000,0.0516400000000000,0.0503750000000000;NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,0.0977590000000000,0.0947190000000000,0.0916060000000000,0.0884740000000000,0.0853270000000000,0.0822110000000000,0.0791400000000000,0.0762710000000000,0.0735360000000000,0.0709190000000000,0.0685620000000000;NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,NaN,0.0994000000000000,0.0959210000000000,0.0925600000000000,0.0894320000000000,0.0864410000000000,0.0834920000000000;]

The code is nothing beyond your original example:

e.g.

IMaxFix2 = inpaint_nans(IMaxFix,num); figure surf(IMaxFix2) title 'Inpainted surface 2'

I have tried num = 1,3,4, and as you suggest in your notes 3 is best, but, by eye, still exaggerates the missing corner points.

If I'm trying to achieve the impossible then don't sugarcoat it, I can take it!

Cheers, Peter

[John D'Errico]

I don't think anyone has ever accused me of sugarcoating things. :)