Increase limit on number of open files

Short question(s):

Is there a way to increase the maximum number of open files allowed at a given time?

For that matter, what exactly sets this limit? Matlab, or the OS? This answer seems to imply that it's the latter; however, I seem to be hitting the limit around 590 files, while my OS says it can handle 1024:

>> system('ulimit -a')
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

Longer explanation:

I know the short answer to this question is "Why on earth would you keep that many files open at once? Rewrite your program to not do that!" So here's the long explanation... I'm open to suggestions, but I'm really trying to avoid a complete overhaul of the code.

I'm running a model that integrates a system of ODEs forward in time. As it runs, output is saved to netcdf files, one time slice at a time. Opening and closing the files for each write is far too time-consuming, so I leave them all open until the simulation finishes, then close them.

In previous versions of this model, all simulation variables were saved to a single file. However, the files have become more and more unwieldy as my research has progressed. Currently, running a set of simulations involves

1) Call the model n times, each of which produces 1 file with nv variables, size nrow x ntime. 2) Postprocess: Loop over all variables, concatenate together across simulations, to create nv files, each holding a nrow x ntime x n array.

At the moment, both steps are comparable in time, taking a few hours to a few days, depending on my chosen simulation time range. I'd really like to get rid of step 2, by creating the individual variable files up front, and having each simulation write to the appropriate time and ensemble-member slice at it computes.

The exact number of output variables varies depending on setup parameters, and I'd like to try to keep things flexible. At most, currently, the model spits out 463 output variables. I often save multiple time-averaged outputs, e.g. one set of outputs with monthly-averages over the entire simulation timespan, and one with daily averages over a shorter period of interest. That would require 926 files to be open at once. And Matlab is none too happy when I try to do that, complaining

Error using internal.matlab.imagesci.nc/openToWrite (line 1169)
Unable to open
/Volumes/MyPassportKak/wceSims/ddelta/testb/testb1_main/exx_40_37.nc for
writing.

Error in internal.matlab.imagesci.nc/openToAppend (line 1236)
                this.openToWrite();

Error in internal.matlab.imagesci.nc (line 124)
                    this.openToAppend();

Error in ncwriteschema (line 87)
ncObj   = internal.matlab.imagesci.nc(ncFile,'a',schemastruct(1).Format);

Error in createoutputfiles (line 95)
    ncwriteschema(file, A);

Error in archivemldata (line 59)
    ncvid = createoutputfiles(Arch.outfile, ...

Error in mixed_layer (line 536)
        [Arch(io).avg, Arch(io).ncvid] = archivemldata(Grd, Arch(io), it,
        datatoarchive, tidx(1));

Error in runmixedlayer (line 115)
            mixed_layer(file, In(iw), pv{:});

And occassionally more cryptically, simply failing silently with a

Caught "std::exception" Exception message is:
blah, blah, something

(Can't remember exactly what the blah, blah was, and can't seem to reproduce that at the moment)

So... any suggestions for getting around the file limit, hopefully without requiring a major rewrite of the code?

[per isakson]

Did you see http://www.mathworks.se/matlabcentral/newsreader/view_thread/319738?

Answer 1

[per isakson]

"Opening and closing the files for each write is far too time-consuming" and "I'm open to suggestions, but I'm really trying to avoid a complete overhaul of the code."

Switching to HDF5 and the High-Level Functions, Easily view, read, and write HDF5 files might be a solution.

• HDF5 and netCDF are similar enough to make the change reasonable
• Open HDF5-files is fast enough(?) so that you don't need to keep the files open

however

• the HDF5-low-level-functions are low level
• the function, h5info, takes time. (The corresponding functionality is included in open of nefCDF(?)). This is not a problem if your program knows whats (and locations) in the file.

[Kelly Kearney]

I ran some very rough speed tests to see if HDF5 would be any faster. This is the first time I've tried working with pure hdf5 files, so my choice of syntax may not be the most efficient (I simply followed the example under the H5D.write documentation for writing a block of variable data to file). However, so far, timings of opening, writing to, and closing the files seem to be on par with netCDF (which makes sense, since netCDF4 uses hdf5, I think).

The figure below shows how the time required to write increases as the number of variables I'm creating increases. The difference in write speed between the keep-open and open/close-per-write versions really start to add up; at 0.14 s vs 1.6 s per write for 400 variables, a 100-ensemble set of 3-year simulations with daily output (109500 writes) would require ~4 hours worth of write time if I leave all the files open while running, but would require almost 2 days worth of write time with opening and closing.

I've attached the code I used to test this. The hdf5 portion is as follows:

folder = 'method5';
mkdir(folder);
file = cellfun(@(x) fullfile(folder, [x '.h5']), var, 'uni', 0);

for iv = 1:nvar
    h5create(file{iv}, ['/' var{iv}], [nz nt]);
end

tic;
for it = 1:nt

      h5_start = [it-1 0];
      h5_block = [1 nz];

      for iv = 1:nvar
          fid = H5F.open(file{iv}, 'H5F_ACC_RDWR', 'H5P_DEFAULT');
          dset_id = H5D.open(fid,['/' var{iv}]);

          mem_space_id = H5S.create_simple(2,h5_block,[]);
          file_space_id = H5D.get_space(dset_id);
          H5S.select_hyperslab(file_space_id,'H5S_SELECT_SET',h5_start,...
              [],[],h5_block);

          H5D.write(dset_id,'H5ML_DEFAULT',mem_space_id,file_space_id,...
              'H5P_DEFAULT',rand(nz,1));

          H5D.close(dset_id);
          H5F.close(fid);
      end
  end
  t(5) = toc;

Is this along the lines of what you had in mind, or is there a more efficient way to use the low-level hdf5 functions?

[per isakson]

It is essential to compare apples to apples. My conclusions regarding netCDF and HDF5 are based on a different use case. It includes thousands of long time series and reading performance is more important than writing. The difference in time to open netCDF4 and HDF5 files manifests with files, which contain thousands of groups and datasets.

Our systems differs. I have

• R2013a 64bit, Win7
• a five years old vanilla desktop
• 8GB ram
• the HDF-data-files on a server disk

I have run your testfiletypes.m and received similar results. (The speed is however a fraction of ours.)

I've added three similar tests, each of which writes to one HDF5-file.

• HDF5 open/close comes from your test suite
• HDF5 low level single is similar to HDF5 open/close, but writes to a single HDF-file without repeated open/close.
• HDF Chunked (the "5" is missing by mistake) writes to one HDF-file with the high level API. The writing is chunked.
• HDF5 high level writes to one HDF-file with the high level API. The writing is not chunked.

I've attached the code. The results are shown in the figure below.

.

The case, HDF5 low level single , might be of interest to your use case. I think however that a single file might cause loss of data if the program errors after a several hours. H5F.flush on a regular basis or split the data between a small number of HDF-files would help.

PS. HDFView is a useful program.

[per isakson]

And a run with the HDF5-data-files on the local hard disk (mechanical). The modified function is attached and the results are displayed in the figure. The case, HDF5 low level single, benefits little from the higher "bandwidth" of the hard disk, whereas the other three more than doubles their speed. Due to open/close the latter cause many more data transfers to the disk.

.

.

Main result

• HDF5 low level single, uses 0.33 sec per write for 400 variables (on my system).
• NETCDF3, keep open, which performs best among OP's cases, uses 0.27 sec per write for 400 variables (on my system).

Discussion

[...]a single file. However, the files have become more and more unwieldy . The HDF5 data file (and CDF4) is organized as a rooted tree in much the same way as a file system, e.g. NTFS of Windows. It is possible to have a one-to-one correspondence between on the one hand the groups and datasets in the HDF5-file and on the other hand the folders and files in a file system. I don't see why a HDF5 file would become more unwieldy than the corresponding system of files. With the help of attributes I recon that the one file HDF5 system could be easier to manage.

The timing reported here does not include the times spent in ncwriteschema and h5create, respectively. These times are significant.

[Kelly Kearney]

Thanks so much for the in-depth response, Per. When I commented on it being unwieldy, I was referring to my original method of output, where I save all variables to one file per ensemble. However, much of my analysis is across ensembles, so I've been constantly juggling arithmatic operations across hundreds of files. So I was looking for a data arrangement that would simplify the analysis.

I suppose in theory I could write everything to one file, whether using hdf5 or netcdf... these would be very large files (~23 GB for my example 3-year-daily-output, 100 ensemble members example), which is why my first instinct was to break it up to one variable per file, but I guess I'm taking up that much space on disc regardless.

So in conclusion, it looks like hdf5 may offer some small organizational advantages over netCDF... but not the speed boost that might tempt me to turn away from the latter, since netCDF is the standard in my field (oceanic and atmospheric modeling). While I tend to favor Matlab for a lot of my analysis, I also use several other tools (ferret, NCO) which are specifically geared towards COARDS-standard (or some variant) netCDF files, so switching away from that would involve a bit of a learning curve for me and may introduce barriers to future collaborations.

Answer 2

[Philip Borghesani]

It is quite possible that you are hitting the system limit and MATLAB is using the other handles. Java tends to keep each jar file open. If your code will run without the Java you could try running -nojvm to free up the file handles used by Java. Otherwise I suggest upping your system limit.

[Kelly Kearney]

Great suggestion! I didn't realize how many files Java was eating up, but so far in my preliminary tests, running -nojvm has indeed prevented the errors.

You suggested upping my system limit if this doesn't solve things in the long run, which brings me back to my original question... how? I seem to get different results if I run ulimit from within Matlab vs. from a terminal window (and for that matter, different terminal sessions give me different results as well), so I'm a bit confused as to whether the OS, Matlab, or some combo of the two is setting the limit. For the record, I'm running Matlab R2013a on Mac OS X 10.9.3.