The compiler is already smart enough to only list referenced assemblies in the final executable that are actually used. No need to fiddle with assembly references or using directives.
The JIT compiler will only ever generate code for methods that are actually called. So you will not have any machine code or compile time overhead due to code that is never used.
Your executable image is getting referenced through a memory-mapped file by the CLR. RAM will only be used if actual content in the DLL is used by the CLR. It depends how the IL of the methods you use is distributed through the image. There are reasonable odds that since the JIT compiler never references the IL, the image data won't be paged into RAM either. In other words, you'll lose some virtual memory space but won't consume a corresponding amount of RAM.
If your DLL is strong named and stored in a non-trusted location then the warm boot time will be slightly longer due to the larger file size.