Making multiple batch calls quickly

6 views (last 30 days)
Stephen
Stephen on 17 Jun 2014
Commented: Stephen on 18 Jun 2014
Hi all,
I have built a GUI which performs 120 function calls at around 5 seconds execution time each. What I've tried to do is send them all out to batch jobs so that I can keep using the GUI while the crunching happens. The problem that I'm running into is that it takes almost as long as the execution time itself to send them all out to batch jobs. Here is the code snippet:
for i = 1:6
for j = 1:20
workerTable{i,j} = batch('lengthyFxn',7,{inputs related to i,j values});
disp(num2str(j));
end
end
My thought process is that I should be able to farm out all of the tasks quickly, which are just calling the same function with different input arguments based on the values of i,j in the loops, and then retrieve them later after the user has taken some time working in the GUI. I added the disp call just to see if anything was actually happening, because this assignment of tasks takes upwards of 5 minutes and the gui and workspace are blocked/unusable in the process. Is there a better or smarter way to do this?
  2 Comments
Edric Ellis
Edric Ellis on 18 Jun 2014
Which cluster type are you using? It's possible that if you're using the 'local' cluster (the default), then you might be able to add the arguments
batch(..., 'AutoAttachFiles', false)
to stop some of the dependency analysis. You might also be better off creating 120 tasks of a single job object, something like this:
j = createJob();
for ...
for ...
createTask(j, @lengthyFxn, 7, {args});
end
end
submit(j)
Stephen
Stephen on 18 Jun 2014
Yes, sorry for not remembering to clarify that - I am using the local cluster. Changing the granularity of the problem by breaking it down into tasks like that did indeed solve the problem I was seeing! What I did was break it down into six jobs assigned 20 tasks each. Thank you so much!!
What is the reason that I'm seeing only one job running at a time? Basically, matlab runs job 1 while 2-6 remain queued, and will only start running the next job when the previous job has finished. Why would this be the case if my cluster has 8 workers? I would ideally like to have all of the jobs running simultaneously.

Sign in to comment.

Answers (0)

Categories

Find more on MATLAB Parallel Server in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!