Looking for the advantages of loading DLLs dynamically as opposed to letting your application load the DLLs by default.
Loading Shared Objects dynamically is the mechanism for allowing plugins ad hoc to running applications. Without plugins a modular application would have to be put together at link-time or compile-time (look at the code of nginx).
Your question is about C#/.NET so in this world dynamic DLL loading requires advanced programming skills. This could compensate all the potential benefits of dynamic DLL loading. You would simply have to write a lot 'low level' code.
In C++/Win32 I often have to load a DLL dynamically when this DLL has some new API function which is not available on older operating systems. In this case I need to ensure the availability of this API at runtime. I cannot just link against this DLL because it will cause application loading errors on legacy operating systems.
As mentioned, you could also have some benefits in a plugin-based environment. In this case you would have more control on your resources if loading DLLs dynamically. Essentially COM is a good example of dynamic DLL handing.
One advantage is for supporting a plugin architecture.
Suppose for example you want to write a service that performs different types of tasks on a scheduled basis. What those tasks are doing, isn't actually relevant to your core service which is just there to kick them off at the right time. And, it's more than likely you want to add support to do other types of tasks in the future (or another developer might want to). In that scenario, by implementing a plugin approach, it allows you to drop in more (compatible by interface) dlls which can be coded independently of the core service. So, adding in support for a new task does not require a new build/deployment of the whole service. If a particular task needs to change, just that dll needs to be redeployed and then automatically picked up.
It also requires other developers to not be concerned with the service themselves, they just need to know what interface to implement so it can be picked up.
We use this architecture for our processing applications to handle differences that our different customers require. Each DLL has a similar structure and implements the same interface and entry method "Process()". We have an XML file that defines which class to load based on the customer and whether there are more methods besides process that needs to be called. Performance should not be an issue until your transaction count gets very high.
If you only load the DLLs you need then the startuptime of the application should be faster.