I don't know to what extent you want to modify your kernel, but if you only are after small "board file" modification, custom drivers etc..., my advice is not to track the whole kernel, but to use the quilt tool, that allows you to maintain your change as a series of patches, and to put your quilt patches under whatever versionning system you want to use.
I am working on several embedded project, as the single developer, and I have to maintain versioning for the various board we are producing. I tried using git, but then I discovered quilt, which is quite rough, but is very handy when you have to maintain only a small set of patches for a varying number of software projetc, each project moving quite rapidly.
Dont put u-boot, the kernel, busybox, every library you are using under versioning. Instead, use some build tool, like ptxdist, build root, etc..., and only maintain quilt series of patch and configuration files for each package.