On all distributions, you can install the vanilla kernel.org sources instead of the distribution-related kernel packages, which is probably a good idea anyway when you want to do kernel development.
However, you'll be in trouble when you want to use any recent distribution with non-2.6 kernels, because they often build libc6 in a way that it cannot run with 2.4. Additionally, a lot of the guts of hardware management (like udev) require fairly recent kernels.
Apart from that, using Debian gives you a barebone system, and installing your own kernels is a breeze with kernel-package.