I am highly doubtful whether it is possible to cleanly hook into the AC_LANG, AC_TRY_COMPILE etc. series of macros without actually rewriting parts of autoconf.
The safe bet for you is to just write a test. Unless you need that test in several projects, you do not even need to wrap the test in m4 macros.
The test would first check for nvcc, then create some test source file and finally try compiling that using $NVCC. Then it needs to examine the results of the compilation (return code and generated files), and finally to clean up any files it might have generated.
Something like
AC_ARG_VAR([NVCC], [nvcc compiler to use])
AC_PATH_PROG([NVCC], [nvcc], [no])
working_nvcc=no
if test "x$NVCC" != "xno"
the
AC_MSG_CHECKING([whether nvcc works])
cat>conftest.c<<EOF
__global__ whatever() {
...
}
EOF
if $NVCC conftest.c && test_whether_output_files_are_ok
then
working_nvcc=yes
fi
rm -f conftest.c conftest.o conftest.what conftest.ever
AC_MSG_RESULT([$working_nvcc])
fi
AM_CONDITIONAL([WORKING_NVCC], [test "x$working_nvcc" = "xyes"])