I have an application (exe) that uses some strongly named dll's. If I understand that correctly, nobody can replace the dll's with their own, prepared versions, because they will not be signed. Nobody can replace the exe too, because it's secured by signing the manifest.
But what if somebody prepares their own exe and changes the signature in the manifest file?
If someone can re-write the manifest file, replace the exe with one of their choice and replace any dll's they want with those of their own choice, then all bets are off.
Bear in mind that the only person able to sign the manifest as you, should be you. If someone else has access to your private key, then they could easily do that. Your users do need to be aware that they should read any information displayed on screen that says What application they're running and Who has signed it. Security is only as good as the weakest link in the chain, sadly that's usually the user.
Have you thought about ClickOnce with the app being downloaded from the source every time it's run, rather than being cached on the users PC. That would at least be a partial mitigation?