Debugging CUDA in Matlab without having to restart

2 views (last 30 days)
Every time I try to launch a CUDA kernel, if the kernel crashes, it seems unrecoverable and I have to restart Matlab. First, the kernel will crash and I'll get this generic error message:
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_UNKNOWN
Not helpful certainly but fine, I'll debug my code. But the error is fatal for Matlab: On my next attempt to merely construct the kernel I get
Error using parallel.gpu.CUDAKernel
An error occurred during PTX compilation of <image>.
The information log was:
<No information>
The error log was:
<No information>
The CUDA error code was: CUDA_ERROR_UNKNOWN.
It also fails to execute a kernel if I supply a pre-built one. My googling has indicated that
>> reset(gpuDevice(1))
will reset things (I have a single GPU, nVidia 650m) but when I do that Matlab segfaults and does not let me continue.
Is there any way to recover from a CUDA crash? Surely it's not necessary to restart Matlab every time a CUDA kernel crashes?
Edit: This is on Windows 8. On a Linux system with a Tesla K20X the same code runs with no problem. The kernel is compiled with
>> !nvcc my_kernel.cu -ptx
in either case.
  3 Comments
Vincent Roberge
Vincent Roberge on 20 Apr 2017
I have the exact same problem using Matlab 2017a. If I call a CUDA kernel with an error, the GPU device becomes busy or unavailable and I need to restart Matlab to use it again. This is really annoying.
covariant_cat
covariant_cat on 20 Oct 2017
Edited: covariant_cat on 20 Oct 2017
Same issue here (2017a). This issue makes debugging CUDA in matlab really annoying. Really hope this to be fixed. I'm surprised that this doesn't get fixed after so many years. Without good support of CUDA, more and more people will switch to python for scientific computing (and deep learning). Mathworks should pay more attention to this.

Sign in to comment.

Answers (1)

Joss Knight
Joss Knight on 26 Dec 2017
If you crash your card, you often need to reload the driver, in much the same way that if a program segfaults, it has to be restarted. Any number of things could go wrong but for the sake of argument let's say the GPU memory containing your kernel instructions has been trashed and is unrecoverable except by being reloaded.
The way to reload the driver is to restart MATLAB. There isn't really any other way to do it because that's the way OSs work.
So really you just need to stop thinking of your CUDA kernel as isolated from the MATLAB executable. When you write your own binaries, whether they be MEX functions or CUDAKernels, you put the entire MATLAB application at risk of your bugs.
  3 Comments
Walter Roberson
Walter Roberson on 27 Dec 2017
The overhead for setting up a separate process and communicating with it is somewhat high. When using a shared gpu instead of a tesla class gpu set to dedicated mode, the overhead would typically be an unacceptable burden.
Joss Knight
Joss Knight on 3 Jan 2018
If you want to do that, start a parallel pool and use parfeval to run your code in your parallel process. If it crashes your card, your client process will be fine.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!