John Robbins wrote about this in his article Do PDB Files Affect Performance?. The simple answer is no (if you compile your release build with both the /optimize+ and /debug switches):
That might be true on other operating systems, but not Windows. If you think they do, then why does Microsoft build every single product they ship with PDB files turned on for both debug and release builds? They wrote the compiler, they wrote the linker, and they wrote the operating system so they know exactly what the effects are. Microsoft has more people focused on performance than any other software company in the world. If there were any performance impact at all, they wouldn't do it. Period. Performance isn't the only thing at Microsoft, it's everything.
Additionally:
When built /optimize+ and a /debug switch, a DebuggingMode.IgnoreSequencePoints is passed to the DebuggableAttribute to tell the JIT compiler that it doesn't need to load the PDB file in order to correctly JIT the IL.
He also has another article entitled PDB Files: What Every Developer Must Know that is also a good read.