How to save workspace variables of a cluster job in .mat file?

Hi,

So I have a script function.m which I need to run using a cluster I have access to.

Within the script I set up the system and it does is a load of calculations. At the end I save the workspace variables (there are 14 altogether I want saved - all in different sized arrays) in a .mat file with a name dependent on the system parameters (set up in within the function's script) that I can go back to later to make the plots I like. Currently this works perfectly fine on my laptop for the smaller systems which it can manage.

Now, when I submit the job in Matlab I do something like this:

c = parcluster('****');
job1 = createCommunicatingJob (c);
num_workers = 12;
job1.AttachedFiles = {'function.m'};
job1.NumWorkersRange = [num_workers,num_workers];
task = createTask (job1, @function, 0, {});
submit (job1)

For the creation of the task in the last two places you put the number of outputs and an array of the inputs. I don't actually have any inputs as I define them all in the code. Also I don't really have any outputs as I only want to save the workspace in a .mat file that I can access later. How can I do this?

When I've tried it with the code above, it submits and runs okay but when I try to find the .mat file it doesn't seem to have saved. Can anyone help me with this?

Thank you in advance!

Answer 1

[Edric Ellis]

If you want to save the outputs to a .mat file, I would recommend providing the full path to where you want it to be saved - otherwise it will be saved in a temporary directory somewhere. (The createCommunicatingJob API that you're using there means that workers start up in a temporary worker-specific directory)

Rather than saving to a file, you could consider making your function return output arguments. Perhaps a struct would be a good way to collect together the 14 output variables. Then, you don't need to worry about saving the results somewhere separate - they are automatically stored with the job object on disk.

I would also recommend looking into the batch function rather than using createCommunicatingJob - it's a bit simpler. To submit an equivalent job using batch, you can do simply

job1 = batch('****', @function, 0, {}, 'Pool', num_workers - 1);

Or, if you modify function to return outputs, you could do:

job1 = batch('****', @function, 1, {}, 'Pool', num_workers - 1);
wait(job1);
outputs = fetchOutputs(job1);

[Elisa Rubbo]

I am trying to work with MDCS on amazon EC2, and I am running into the same problem: I don't know how to specify the full path for the variables to be saved (I just put '/shared/persisted/filename.mat', but didn't work), so I cannot access the files I saved.

Would you know where I can find the full path I need to specify?

Thanks!