Tip of the Week - Create a Pool of CPUs and GPUs to Speed Up Hyperparameter Tuning for Deep Learning

General
Follow


Tip of the Week - Create a Pool of CPUs and GPUs to Speed Up Hyperparameter Tuning for Deep Learning

Aycan Hacioglu on 27 Aug 2021
Latest activity Reply by Aycan Hacioglu on 27 Aug 2021

Nowadays, many instructors are integrating AI to their courses. In a distance learning setting, the hardware students use to train their models vary. Training time of the deep learning models can be shortened with a pool of GPUs, CPUs or a pool of CPUs and GPUs locally or in the cloud. Accuracy of the results can depend on the hyperparameters used to train the models.

In MATLAB, Experiment Manager (introduced in R2020a) makes it easy to train networks with various hyperparameters and compare the results. Different models can be run in parallel by clicking on “Use Parallel” button in Experiment Manager App. But what if your computer has multiple CPU cores and a GPU? Would you be able to use Experiment Manager with a pool of CPUs and a GPU? The answer is “yes”. For example, my computer has 1 NVIDIA GPU and an 8 core CPU. To use all these computational resources, I typed these lines in my command window in MATLAB:

parpool(9);
spmd
if labindex==1
gpuDevice(1); %select GPU on worker 1
else
gpuDevice([]); %deselect GPU on remaining workers
end

Then, I clicked on “Use Parallel” button in Experiment Manager and used a modified setup function in Experiment Manager to change the execution environment from CPU to GPU:

if isempty(parallel.gpu.GPUDeviceManager.instance.SelectedDevice)
options=trainingOptions(~,"ExecutionEnvironment",'cpu');
else
options=trainingOptions(~,"ExecutionEnvironment",'gpu');
end 

Default training options automatically use an NVIDIA GPU if there is one, and specific hardware can be selected using “ExecutionEnvironment” in the trainingOptions.

If you like to learn more about the fundamentals of parallel computing, check out “Parallel Computing Fundamentals” on our documentation and gain some hands-on experience with Parallel Computing through "Parallel Computing Hands-On Workshop" .

Sign in to participate
Aycan Hacioglu
Aycan Hacioglu on 27 Aug 2021

A few things to consider if you create a pool of CPUs and GPUs:

  • CPU and GPU performance (if your GPU is way faster than your CPU for the same task, the GPU might end up doing bulk of the work)
  • How many experiments you have to run in Experiment Manager
  • Precision of the computations (Experiment Manager doesn’t show which experiments are run on which hardware. There might be slight differences in the results run on CPU vs. GPU)
  • Communication overhead between the workers (is your problem large enough to benefit from parallelization)
Go to top of page