I have to give a +1 to Drejc. The single most reason why not to refactor is if you don't have unit tests. But even that doesn't mean you should not refactor. I should say that my modus operandus is code functionality first and then always refactor to deal with design-debt. Repeat ad-nauseaum.
The "good enough" pattern is probably the most used programming methodology on earth, and that's one reason why our software is buggy and messy. Instead of chosing the least energy "path", we always choose the least energy "step".
But, beware! Refactoring without the proper tools (and this will depend on the language and IDE) is like hammering a screw: you can get there, but it won't be effective (and you can get in trouble).