Matlab loads part of huge file

Good day to all.

I'm loading a huge file of data. First row contains station name. First column contains dates. The data are of type double. The csv file i'm loading haw a total of 35065 rows and 5000 columns. I use the following code:

    TestData=importdata (filename,';');
    vars = fieldnames(TestData);
    for i = 1:length(vars)
        assignin('base', vars{i}, TestData.(vars{i}));
    end

Matlab loads only 17504 rows and ignores the rest. Im working on a relatively powerfull linux machine, with an i7 processor and 24G memory.

Does anyone know the nature of the problem here. Is it matlab limitaion? Can i do something to fix it?

Thank you all!

[Stephen23]

Do not create or use variable names dynamically like this. Use a cell array or structure instead:

http://blogs.mathworks.com/videos/2010/03/08/top-10-matlab-code-practices-that-make-me-cry/

http://www.mathworks.com/matlabcentral/answers/143-how-do-i-make-a-series-of-variables-a1-a2-a3-a10

Answer 1

[per isakson]

My guess is that row number 17504 of your file contains something that causes importdata to stop reading.

A little experiment

    >> importdata( 'cssm.txt' )
    ans = 
              data: [4x5 double]
          textdata: {'A'  'B'  'C'  'D'  'E'}
        colheaders: {'A'  'B'  'C'  'D'  'E'}
    >> ans.data
    ans =
        17    24     1     8    15
        23     5     7    14    16
         4     6    13    20    22
        10    12   NaN   NaN   NaN
    >>

where cssm.txt contains

       A     B     C     D     E
      17    24     1     8    15
      23     5     7    14    16
       4     6    13    20    22
      10    12   BBB    21     3
      11    18    25     2     9

"BBB" in the 4th data-row causes importdata to

1. stop reading
2. pad that row with NaN and
3. skip the remaning rows

What is worse is that importdata does this without warning the user.

Answer 2

[thankar]

Hello again.

Line 17504 does not contain anything wierd! I checked. However, i have to add that i tried another solution that worked. I split the input file, making files with 1000 columns maximum. Everything was loaded correctly. So it seems, that the problem was solved in a way. However the question still stands! Is it a memory limitation on matlab, when loading data?

Sakis

[per isakson]

I assume you have 64bit system.

    >> 35065 * 5000 * 8 /1e9
    ans =
        1.4026

That's 1.4GB and another <3GB for the file itself.

Sounds a bit strange. I would like to make a test. Could you you upload the file somewhere?