Because you don't need to. A Visual Studio license is pretty expensive, so having one just lying around on a server where no one's using it is just a waste.There are a couple of arguments why you would still need to install a full blown Visual Studio instance on your Continuous Integration server - but here are their counter arguments:
Reason 1: I need it to compile.
Reality: No, you don't. You need MSBuild to compile, but that is available for free, in the Windows SDK. Note that there are several versions for different operative systems and .NET versions, so be careful to download the correct one.
Reason 2: I need it to make quick fixes on the server.
Reality: No, you don't. You shouldn't make quick fixes on the server - you should check out from your version control system, make the fix, build and run tests locally until it works, check in, and have the CI system do the rest for you. That's why you have a CI system.
Reason 3: Without Visual Studio, I can't run MSTest no my CI server.
Reality: Wrong. AFAIK, the MSTest runner is also part of the SDK (at least that's what it seems like on our CI server here - although I can't verify it since we don't have any tests at the moment...). However, a quick googling found this blog post which explains how to do it without the SDK as well. I haven't read through it in detail, so I can't promise that it works, or that it's legal. You have been warned.
Feel free to add more reasons in comments, and I'll counter them.