I've been teaching a class on unit tests, refactoring and so forth, and this is probably the thing that most people get wrong. Refactoring is not just changing the code. It is changing the code without changing the external functional behavior. That is a very important point.
In other words, you need to have some way to verify that the external functional behavior is intact after the refactoring. Lacking divine insight I find unit tests very useful for that. In his book on Refactoring, Martin Fowler stresses the use of automated tests for this verification.
If your code was developed using TDD you will have the necessary test suite as it is developed during the development of the code itself. If you need to refactor code for which no tests are available, your best approach would be to set up automated tests before you make any changes to the code. I realize that setting up tests for existing code can be hard, but you will learn a lot about the code while doing so.
You may also want to check Bruce Eckel's essay on strong typing versus strong testing as it discusses the feedback you get from the compiler versus the feedback you get from your test suite.