Speaking as a developer then I would recommend leaving as much in as you possibly can get away with. The reasoning?
One day, you will encounter a bug in your program where the ONLY piece of information you have is a stack trace, and the bug cannot be reproduced on command, it was completely unanticipated by the original programmers, and it is YOUR job to fix it. The more information available to you in that stack trace the better! Leave all debug information in!
If you can, then use a logging framework (to get the stack traces to a file) which can provide information about the jar-file in which each class was found. Logback can do this, and I believe log4j can too.
You may not be allowed to include all this information, but I believe you should first yell and scream and say that it should be left in for contingency reasons.
Performancewise I believe that since HotSPot it hasn't mattered.