Not able to kill bad kernel running on NVIDIA GPU

Question

Hi,

I am in a real fix. Please help. Its urgent.

I have a host process that spawns multiple host(CPU) threads (pthreads). These threads in turn call the CUDA kernel. These CUDA kernels are written by external users. So it might be bad kernels that enter infinite loop. In order to overcome this I have put a time-out of 2 mins that will kill the corresponding CPU thread.

Will killing the CPU thread also kill the kernel running on the GPU? As far as what I have tested it does'nt.

How can I kill all the threads currently running in the GPU?

Edit: The reason I am using CPU threads that call the kernel is because, the sever has two Tesla GPU's. So the thread will schedule the kernel on the GPU device alternatively.

Thanks, Arvind

Answer 1

Will killing the CPU thread also kill the kernel running on the GPU? As far as what I have tested it does'nt.

Probably not. On Linux u can use cuda-gdb to figure that out.

I don't see the point of sending multiple kernels to the GPU using threads.. I wonder what happens if you send multiple Kernels to the GPU at time.. Will the thread scheduler of the GPU deal with that?

Answer 2

It doesn't seem to. I ran a broken kernel and locked up one of my devices seemingly indefinitely (until reboot). I'm not sure how to kill running kernel. I think there is a way to limit kernel execution time via the driver, though, so that might be the way to go.