I come from the mindset that servers should be designed to perform a specific task. An Active Directory Controller, for example, shouldn't double as a SQL sever.
From the prospective of resources available, it becomes difficult to balance the load between multiple server applications. A build server performing a build, for example, would affect performance of any other services on the box. It's likely that your build server will be used to build many different projects. Compilation and the plethora of tasks associated with a typical build (e.g., code analysis, unit testing, etc.) are expensive and CPU/disk intensive operations.
Additionally, have you ever noticed how your desktop becomes slower and less stable through time? This is often due to the of the number of applications that you install. Added dependencies and increased memory consumption will negatively impact the reliability of your servers.
Servers should be left clean and be designed to perform their specific task with as few interruptions as possible.