Parfor vs CreateMatlabPoolJob(): is there a way to avoid repeated overheads?

Hello everyone,

I am running the following code on a 8-core computer:

pjob = createMatlabPoolJob();
set(pjob, 'FileDependencies', {'test_parfor.m'});
createTask(pjob, @test_parfor, 1, {});
submit(pjob);
waitForState(pjob, 'finished');
results = getAllOutputArguments(pjob)

test_parfor() is a function that runs a parfor loop for testing purposes, i.e.

function [elapsedTime] =  test_parfor()

N = 5000;
A = zeros(N,1);
tic;
parfor i=1:N
    E = eig(rand(100))+i;
    A(i) = E(1);
end
elapsedTime = toc;
disp(elapsedTime);
end

As you can see, the function test_parfor() returns elapsed time. The good news is that I obtain a speedup almost linear with respect to the serial case (ca. 8x). The bad new is that the overhead is huge (15 seconds!). Specifically, when the job is submitted to the local scheduler, it stays pending for 15 seconds before starting running. As far as I understand, the overhead is the same of opening matlabpool with

matlabpool open 8

Actually, if I open Matlabpool and simply run the function with a parfor, I get the linear speedup - because I can open matlabpool at the beginning, then run as many parfor loops I want. As far as I understand, every time I submit a job to the scheduler, it automatically opens Matlabpool and generates the overhead.

Is there any way to avoid the overhead using createMatlabPoolJob() every time I submit a job (so that the overhead is there only once, and not for every submission to the scheduler)? Maybe something similar to what I do when I open matlabpool once, and then run as many parfor loops I like.

I apologize for the long question. Best, Roberto