There are levels and levels of refactoring.
I constantly refactor to remove duplication. I don't go looking for duplication, but if I see it, I fix it. I'm constantly using Extract Method to make methods smaller, and following all ReSharper suggestions or warnings (I configure ReSharper to make anything I'm likely to ignore into a Hint). This is also a benefit because you get used to seeing the nice cheerful green square at the top, telling you all is good. When it's a different color, you know something's wrong.
Larger refactorings (the kind the OP may be concerned about), I largely treat like I would a new feature. I'll try to get consensus on how important it is, and will schedule it accordingly. I will not make such a change unless the code involved has a high degree of unit test coverage. "If it's not tested, then it's broken" is one corollary to Murphy's Law. Also, if it's already broken, then I'm not going to make it worse by refactoring poorly-covered code.
I also do not refactor to make the code fit some pattern, or to be more easily modified for the future. When the future arrives, then it will arrive with some failing unit tests that won't succeed unless I make the changes. If there are no such tests, then I can't tell if my changes broke anything, so I won't refactor.