How do I debug a CUDA library with only 1 graphics card running X11

Question

I'm running a CUDA library that I need to debug for memory problems and other issues. But when I attach cuda-gdb to the process I get the error

error: All CUDA devices are used for X11 and cannot be used while debugging.

I understand the error, but there has to be a way that I can debug the issues. Since I only have 1 GPU, it really isn't practical to turn off X11.

On non Nvidia hardware I thought there was a way to emulate a cuda gpu. could this be setup for debugging even though I have an NVIDIA gpu?

Answer 1

First of all, as you are using Linux you're in a lucky position as you can kill X pretty easily just for the time of debugging.

However, if you really want to stick to running X while debugging you are out of luck, as this is not possible for a very good reason: the display driver has a protection mechanism called watchdog timer which is enabled when the GPU in use also drives a display. The watchdog timer interrupts any kernel that runs for longer AFAIR 5s. This is intended to prevent GPU lockups.

Alternatively, you could try using ocelot, but I am not sure how good are the debugging features it provides.