We install the libraries on our machines (for the Visual Studio toolbox support), but also copy the binaries to our subversion repository:
/trunk/thirdparty/[company]/[software]/[version]
Once a new reference is added (as a result of dragging a new control to a form), we change the reference to point to the binary in the working copy instead. This means they are there on a full checkout working copy and are referenced by relative paths, which therefore works on the build server without installing the control libraries there.
It also allows us to build software that uses old versions of control libraries while new code written can use new ones.