Not necessarily - if the dependencies have not changed then they will not be rebuilt. If you select "ReBuild" then Visual Studio will rebuild the dependencies as well but you will find that a normal build will reuse the existing dependency assemblies if the source code for those assemblies is unchanged.
C# uses an assembly's metadata to build against. This metadata is a part of the assembly itself and as such the entire assembly must be present at compilation time for the compiler to resolve any external types and their members.