Time based moving average - is my approach correct?
10 views (last 30 days)
Show older comments
Hello Matlab folks,
I'd like to get a critique for my code because I am not sure if I am right. First of all, I have a set of data like
735728,605428241 80,6717800000000 11,6650710000000
735728,605451389 80,6717800000000 9,12566720000000
735728,605474537 80,1416200000000 9,24074830000000
735728,605497685 80,1416200000000 9,20919460000000
735728,605520833 80,1416200000000 9,47081590000000
735728,605543982 80,3848000000000 9,28115980000000
735728,605567130 80,3848000000000 8,55264740000000
735728,605590278 80,3848000000000 9,47107710000000
735728,605613426 80,5608400000000 9,42203640000000
735728,605636574 80,6196400000000 9,32575490000000
735728,605659722 80,6196400000000 9,25049830000000
735728,605682870 80,6998100000000 9,48914560000000
735728,605706019 80,6998100000000 9,52505890000000
735728,605729167 80,6998100000000 9,12462120000000
The first row is the time in format 'dd.mm.yyyy hh:mm:ss'. The sample rate is two seconds. Now I want to plane that data to get it a bit smoother and I want to do this by using a moving average of 60sec. To do so, I found the function movavg to be correct. I ended up with the following code
clc
clear all
close all
% Get the data
filepath=''; % Note: There is a path in the original code
fprintf('Path: %s\n', filepath)
fprintf('\nLoad totaldev ...\n')
totaldev = xlsread(strcat(filepath,'totaldev.xlsx'));
fprintf(' ...done!\nLoad plant_load ...')
plant_load = xlsread(strcat(filepath,'load.xlsx'));
fprintf(' ...done!\n')
data=[plant_load(:,1),plant_load(:,2),totaldev(:,2)];
data(:,1)=x2mdate(data(:,1));
fprintf('Prepare data ...\n')
% Remove values below 0 and above 100 (plant load is 0% to 100%
data(any(data(:,2)<0,2),:)=[];
data(any(data(:,2)>100,2),:)=[];
% Remove values below -100 and above 100 (error can't be more than +-100%)
data(any(data(:,3)<-100,2),:)=[];
data(any(data(:,3)>100,2),:)=[];
% Compute average of time series
% Sample rate is 2sec, so 30 values cover a minute
data_avg=[data(:,1), movavg(data(:,2),30,30), movavg(data(:,3),30,30)];
fprintf(' ...done!\n')
fprintf('\nCreate plot')
% Plot 'em
plot(data(:,1),data_avg(:,2),data(:,1),data_avg(:,3))
datetick('x','keepticks','keeplimits');
I'd like to know now if I did a mistake or if the average is wrong. Maybe someone can help.
Cheers from Germany
10 Comments
Cedric
on 26 May 2014
Edited: Cedric
on 26 May 2014
Hi Star Strider,
Ok. I'll think more about it, but my guess is that most of the usual methods will fail to produce a correct block average when there are missing data/entries. One solution could be to build a minuteID (=1 for the first n<=29 entries, =2 for the next 29 entries, etc ) before removing invalid entries, and then to use ACCUMARRAY on remaining entries, using the minuteID as a block ID..
Star Strider
on 26 May 2014
Were it my data, I’d isolate the one-minute segments, do the thresholding on the selected block of data, compute the mean of the rest, and continue on. That just seems to me to be the easiest way to do the processing. I assume the data and times are continuous in every record.
I am prepared to stand corrected...
Accepted Answer
Star Strider
on 26 May 2014
My pleasure!
If you only need to run the loops once, saving the results of the one-minute-means in a ‘.mat’ file, a loop isn’t much of a problem since you only have to do it once.
Assuming your matrix to be X, it’s probably as easy as:
X = rand(180,3);
for k1 = 1:fix(size(X,1)/30)
for k2 = (k1-1)*30+1
Xm(k1,:) = mean(X(k2:k2+29,:));
end
end
That runs without error, with Xm storing the one-minute means. Be sure to use the save function to save the averaged matrices, then use load to retrieve them when you need them. Then you only have to do the loop calculation once for each data set.
4 Comments
Star Strider
on 27 May 2014
I completely agree about being passionately curious about science. I pity those who are not. Being too curious is fine. Being selective and focused about your writing is also.
My applicable background here is in biomedical engineering (largely electrical), statistics and biomedical signal processing. So I’m sensitive about filters that aren’t designed for a specific purpose. They can do unkind things to one’s data.
Have fun!
More Answers (2)
Marc
on 25 May 2014
1 Comment
Star Strider
on 25 May 2014
I believe you are seeing ‘aliasing’. I don’t use moving average filters because they have undesirable frequency characteristics. If you want to filter out noise, I suggest a low pass filter.
You will likely need to do a Fourier transform on your data (if you have not already) to see its frequency content and make the appropriate filter choices. The Signal Processing Toolbox makes filter design and implementation quite easy.
Marc
on 25 May 2014
2 Comments
Star Strider
on 25 May 2014
If you want the average over each minute, the moving average filter is not your optimal choice. It takes the mean over samples 1:30 then 2:31, 3:32, etc. If that is what you want, use the moving average filter in spite of its suboptimal passband characteristics. (I don’t have the Financial Toolbox, so I can only simulate what I believe the movavg function does. It is not well-documented from a signal processing perspective.)
I infer by a ‘ 1 min average ’ that you want the average of the first minute (samples 1:30), the second minute (samples 31:60), etc. The documentation for mean has all the information you need to do that.
See Also
Categories
Find more on Signal Generation and Preprocessing in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!