How do I use an augmentedImageDatastore with DispatchInBackground?

I am working on a deep learning CNN problem. I want to use the augmentedImageDatastore to preprocess my images. However, the augmentedImageDatastore is slower than preprocessing the images before hand and storing them on the disk. To speed up the computations, I want to set the DispatchInBackround property to true.

However, when I do this, a parallel pool is created on my machine. Whenever a parallel pool is active, Matlab complains that my GPU device is busy and will refuse to train the network on the GPU (also when the ExecutionEnvironment is set to 'auto', 'gpu', or 'parallel').

I am using Matlab 2018a, with a 960m GPU and i7-6700HQ processor. The number of threads per worker is 1 (I have also tried 2 threads per worker), with 4 workers per pool (I have also tried 2 workers per pool).

The exact error is: Error using trainNetwork (line 154) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable

Error in cnn (line 150) convnet{nested_idx} = trainNetwork(train_Store, layers, options);

Caused by: Error using gpuDevice (line 26) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable

Or, when using parallel (and the pool is started automatically): Starting parallel pool (parpool) using the 'local' profile ... Preserving jobs with IDs: 1 2 3 4 5 6 7 because they contain crash dump files. You can use 'delete(myCluster.Jobs)' to remove all jobs created with profile local. To create 'myCluster' use 'myCluster = parcluster('local')'. connected to 4 workers. Error using trainNetwork (line 154) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable

Error in cnn (line 150) convnet{nested_idx} = trainNetwork(train_Store, layers, options);

Caused by: Error using nnet.internal.cnn.util.GPUShouldBeUsed (line 22) Error detected on worker 2. Error using gpuDevice (line 26) An unexpected error occurred during CUDA execution. The CUDA error was: all CUDA-capable devices are busy or unavailable

[Joss Knight]

It looks like your driver is configured to only allow one process to access the GPU, which isn't surprising if this is a Windows laptop. Do you have the same issue when you run two instances of MATLAB - is only one of them allowed access to the GPU?

I'm a little surprised that DispatchInBackground can cause this issue. Try starting your pool in advance and disabling the GPU to force the pool to work on the CPU:

parpool(4);
spmd
    setenv CUDA_VISIBLE_DEVICES -1;
end

[Alpha Bravo]

Do you know how to configure the driver so that multiple processes may use it? As you thought, multiple Matlab instances cannot run with the GPU at once - they both fail to access it if they compete for it. I put the code before I created the augmented datastores, but it had no effect. The error still occurs. :(