It's like the National Parks -- Always leave it a little better than you found it.
To me, that means any time I open code, and have to scratch my head to figure out what's going on, I should refactor something. My primary goal is for readability and understanding. Usually it's just renaming a variable for clarity. Sometimes it's extracting a method -
For example (trivial), If I came across
temp = array[i];
array[i] = array[j];
array[j] = temp;
I would probably replace that with a swap(i,j) method.
The compiler will likely inline it anyways, and a swap() tells everyone semantically what's going on.
That being said, with my own code (starting from scratch), I tend to refactor for design.
I often find it easier to work in a concrete class. When its done and debugged, then I'll pull the old Extract Interface trick.
I'll leave it to a co-worker to refactor for readability, as I'm too close to the code to notice the holes. After all, I know what I meant.