Is it practically possible for a malicious developer exploit your application by swapping a genuine .dll file for a modified one.
If so are there steps that can be taken to protect an app. Are there some types of functionality that should not be placed in .dlls for example?
For .NET applications, you can sign the DLL. I'm not sure if something similar exists for non-.NET assemblies.
As long a signature of the exports is known is very easy to replace a DLL. It's also possible to reverse engineer the signature if it is not known.
To help prevent this problem you can use Code Signing, so you can verify the signature before using the DLL.
Anyone with write permissions to your application installation directory will also be able to modify your executable file, so trying to protect against DLL substitution is far from sufficient for many attack scenarios. Is there some particular scenario about which you are concerned?
You should also consider DLL injection, where by a malicious user can for example view passwords in your password fields.....
check out these links:
http://en.wikipedia.org/wiki/DLL%5Finjection
http://www.codeproject.com/KB/DLL/DLL%5FInjection%5Ftutorial.aspx