Your application shouldn't be noticeably slower, however if you leave too many objects on the finalizer queue the GC can cause momentary breaks in execution (the .Net GC isn't async). Technically .Net should be as fast as native code, but I have read a few times that it isn't: but at the same time it's only marginally slower (relevant only, as they say, in academic or real time situations).
As for the longer startup time you can ngen your assemblies. This pre-compiles the assemblies into native code so you don't incur the cost of a JIT compilation when a method is accessed for the first time (remember, JIT only compiles the first time a method is invoked and never again while the application is running).
I don't think the .Net runtime should be a problem: especially if you target .Net 2.0. Windows XP SP2 came with .Net 2.0 so any responsible customers shouldn't have any problems.
And once they have the framework on their PC the programs you compile are TINY (it's actually depressing when you write 1000s of lines of code only to get a DLL that is less than a meg). If you use one-click deployment the cost of updating is also pretty small (if you are disciplined and shell functionality out into appropriate assemblies).