About Friedman test implementation

18 views (last 30 days)
Hi,
I found (maybe I am wrong) that the Matlab implementation of the Friedman test does not give me the same results than a "classical" hand-made resolution for a problem.
I have used the dataset of the Table 6 of the paper "Statistical Comparisons of Classifiers over Multiple Data Sets" (Demsar, Journal of Machine Learning, 2006).
Even the mean ranks of the different classifiers (columns) aren't the same. From that moment, chi-square value does not agree, etc...
Does anybody know why are these differences? Best regards.
  2 Comments
the cyclist
the cyclist on 21 Feb 2012
You might have a better chance of someone exploring this question if you pasted the data from that paper here, in a way that someone could just directly put it into MATLAB.
José Manuel
José Manuel on 21 Feb 2012
You are right.
data=[0.7630, 0.7680, 0.7710, 0.7980; 0.5990, 0.5910, 0.5900, 0.5690; 0.9540, 0.9710, 0.9680, 0.9670; 0.6280, 0.6610, 0.6540, 0.6570; 0.8820, 0.8880, 0.8860, 0.8980; 0.9360, 0.9310, 0.9160, 0.9310; 0.6610, 0.6680, 0.6090, 0.6850; 0.5830, 0.5830, 0.5630, 0.6250; 0.7750, 0.8380, 0.8660, 0.8750; 1 , 1 , 1 , 1 ; 0.9400, 0.9620, 0.9650, 0.9620; 0.6190, 0.6660, 0.6140, 0.6690; 0.9720, 0.9810, 0.9750, 0.9750; 0.9570, 0.9780, 0.9460, 0.9700];
And I add this question:
Regarding to the "two categorical factors, what is the meaning of this phrase os the help? "Friedman’s test is a nonparametric test for data having a two-way layout (data grouped by two categorical factors)." I noticed that all the examples of the help are with only two factors in the row dimension (with several reps)...

Sign in to comment.

Accepted Answer

the cyclist
the cyclist on 21 Feb 2012
I notice a couple things. First, that paper is doing the ranking from largest to smallest effect. MATLAB is doing the opposite. This is just a matter of convention, and should not effect the final results. From MATLAB, I get
meanranks: [1.8571 3 2.0714 3.0714]
The paper gives [3.143 2.000 2.893 1.964].
If we "invert" the MATLAB result, it would be
[3.1429 2.0000 2.9286 1.9286].
These results are (almost) the same as the paper.
I'm not sure, but the remaining differences is likely due to how tied ranks are handled. My advice would be to open up the file ( edit friedman ), place a breakpoint at the beginning, and step through the code line-by-line to see how the exact ranks are calculated.
  5 Comments
the cyclist
the cyclist on 21 Feb 2012
No, I do not agree. I believe that MATLAB is applying a correction term that is not in the paper. In the code for friedman() you'll see these lines:
% Get a matrix of ranks. For the unusual case of replicated
% measurements, rank together all replicates in the same row. This
% is the advice given by Zar (1996), "Biostatistical Analysis."
m = X;
sumta = 0;
for j=1:r
jrows = reps * (j-1) + (1:reps);
v = X(jrows,:);
[a,tieadj] = tiedrank(v(:));
m(jrows,:) = reshape(a, reps, c);;
sumta = sumta + 2*tieadj;
end
I am not familiar the reason this is necessary, but if you look at the calculation of chistat inside friedman(), you will see this correction applied when "sumta" is not equal to zero. That is what makes the MATLAB result different from the paper.
José Manuel
José Manuel on 22 Feb 2012
You are right. I also do not know why Matlab does that.
Thank you very much.

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!