MODEST

Maximum Of Difference Enlarged by Smoothed Timeseries is to detect and correct outlier phases of timeseries data or images.

MODEST Help

****************Introduction****************

Timeseries data encounters outlier phases other than a single outlier value or separate outlier values from time to time, impacting their further usage. Its adjustment needs determining the onset and end of the phase first, and is thus more challenging than smoothing a single outlier value.

Even though there prompts several moving and detecting methods, like moving average, moving median, RLOWESS, coupled with 3-sigma principal, unfortunately, still no handy tool exists to cope with this problem.

Here we developed a software based on an effective combination of existing methods, and named it Maximum Of Difference Enlarged by Smoothed Timeseries (MODEST) to detect the onset and end of the outlier phases and correct them. MODEST can be applied once or recursively as users' designation, where original difference will be substituted by the smoothed difference at detected points and other timeseries differences remaining the same. Apart from ordinary form of timeseries, MODEST can also be applied to images with temporal information pixel by pixel, typically remote sensing images.

****************Easy Start****************

Case 1

1. Click "Try random" with default setting and visualize them.

2. Click "MODEST" with default setting and visualize the adjusted data with also parts of the process.

3. Change "sample size", "exagerate_n", "x_param", or "y_param", and repeat 1-2.

You can see the robustness of Static HL-Blended Near Comp. MODEST.

Also, you can change other settings or values to justify its PROs and CONs.

Case 2

(Click "Clear" to clear existing plots if Case 1 was just implemented)

1. Click "Import" button (Green arrow in menu), select "modest_try.xlsx", and you can see the plots of the data.

2. Click "MODEST" with users' setting, and you can visualize the adjusted data with also part of the process.

3. Click "Export" button (Red arrow in menu), and you can save the adjusted data as designated.

* Click "Reset" button (Red dot in menu) to reset this app if necessary.

Case 3

(Click "Clear" to clear existing plots if other cases were just implemented)

1. Click "Import" button (Green arrow in menu), select "modest_mc.tif", and you can see the annual nighttime light pattern of the Macau, China.

2. Click "MODEST" with users' setting, and you can visualize the adjusted images, with changed boundaries plotted (can be unplotted); red/blue edges delineate increased/decreased areas.

3. Tick/untick "Edge" to show/unshow the boundaries of adjusted areas.

4. Click "Pixel Curve" to sample some Points Of Interests (POI) to illustrates pre-adjustment and post-adjustment dynamic curves.

5. Click "Area Curve" to select Area Of Interests (AOI) to illustrates pre-adjustment and post-adjustment dynamic curves within it.

button (Red arrow in menu), and you can save the adjusted images as designated.

* "Export", "Reset" buttons are functional as in Case 2.

****************Quick View****************

This section quickly introduce the interface of MODEST.

——Button Usage

1. Try random: Generate random curve samples as users' setting and plot them in Visualization panel.

2. Clear: Clear all plots in Visualization panel.

3. MODEST: Adjust randomly generated or imported data using MODEST as users' setting.

4. GreenArrow: Select one file to be adjusted. (xlsx, .csv, .xls are supported).

5. BlueArrow: Export adjusted data to user's designation as a CSV file.

6. RedCircle: Reset this app.

——Parameters Setting

1. MODEST param.:

1.1 "Static" would apply MODEST once to curves with outlier phases. "Adaptive" would recursively apply MODEST until the standard deviation of the adjusted curve is smaller than appointed one curve.

1.2 "High Only" asks MODEST should only adjust higher outlier phases, while "HL Blended" asks MODEST to adjust both. Sometiems, "High Only" is more useful in adjusting highly mess piece-wise functions.

1.3 "Near Comp." sets the baseline of standard deviation comparison is the nearest standard deviation in previous loop, and "Origin Comp." sets the baseline as that of the original data.

2. MyRLOWESS:

RLOWESS is used to smooth the difference of original difference of timeseries.

2.1 "frac" determines how many percent of data would be included to generate local smooth; too small would fail to get smoothed data, while too large would make MODEST to sensitive.

2.2 "times" determines how many times of loop would be taken in RLOWESS process.

3. Timeseries:

In randomly generated timeseries, parameters here determine the year of start and end, piece numbers (only useful in piece-wise generation), and year interval.

4. Random sample generation:

4.1 "sample size" determines how many bunch of testing data would be generated.

4.2 "exagerate_n" determines to what extent the random noise will be added to defined functions.

4.3 "x_param" determines the period of outlier phases.

4.4 "y_param" determines the extent of the positive or negative offsets.

4.5 "loop_param" determines the loop times of recursively applied MODEST in each curve.

5. Other interacts

5.1 Variable Name: The name of imported data, which will be shown in data legend and set as default name in "Export...".

5.2 Processing: States of process would be recorded here.

Please apply MODEST to different real-world cases, and contact us if any problem.

Hope you can witness and promote the power of MODEST.

Best wishes!

Developers

2022.8