views:

534

answers:

12

Possible Duplicate:
How do you manage to write high quality code very quickly?

When emergency code fixes are required ASAP, how do you ensure that the code you're writing is better than the code you replaced? What processes do you keep when time is of the essence?

A: 

It depends on what you mean by better. You can write unit tests to cover the code you are fixing to make sure the code is working. Then you can rollout and refactor later.

If better means its cleaner, well factored. Well that's subjective. If better means faster you should set up an unit test to verify that.

I guess the process that you should keep above all is testing.

John Nolan
"You can write unit tests", or just "You can write tests"? IMO you could write new system/acceptance tests, instead of (or as well) as new unit tests.
ChrisW
acceptance tests / unit tests both are good.
John Nolan
I find some people write as though unit testing were the only kind of testing (and/or the only kind of automated testing).
ChrisW
think that is evolving with the rise in popularity of BDD.
John Nolan
@ChrisW I find that my organization wants a System test for every unit test. It's a pain.
C. Ross
@C. Ross - [I prefer system tests](http://stackoverflow.com/questions/856115); unit tests are a necessary evil/substitute, if system testing is expensive, or if debugging-during-integration is expensive, or if other system components (against which I'd want to test my new component) aren't available.
ChrisW
+3  A: 

What processes do you keep when time is of the essence?

Testing: new code should pass all the same (automated, regression) tests as previous code, plus the new tests which you've written to test the new feature/bug fix.

ChrisW
+16  A: 

The simple answer is that you don't write quality code under duress so you must do everything you can to avoid or mitigate the situation.

If anything these are the times when you need to adhere even more strictly to good development practice. What's worse, leaving an application with an error for a few more hours/days (or perhaps taking a site off line if necessary) or replacing one problem with another which may only manifest itself days or even weeks later?

The management need to see that a rush job will only store up problems for the future - explaining this in terms of the revenue lost would be a good place to start - so that they give you the time and space to not be under duress.

At the very least you need to subject any emergency fix to a full suite of regression tests and perform rigorous QA on it after the patch has gone live and be prepared to re-patch if the fix is found to be wanting.

ChrisF
Sounds good, but when the cost of filing taxes late is measured in the millions of dollars, it's difficult to convince anyone to wait. The cost of the bug you might introduce is almost certainly less than the cost of being late.
C. Ross
ChrisF
@C Ross, so what happens when the bug you introduce because of a rushed fix is contributes to being even later?
Ash
+4  A: 

Unit tests. Unit tests ensure that one change in functionality doesn't impact other areas. Other than that I have to agree with ChrisF, rush jobs are never good.

wheaties
+1  A: 

This depends on the circumstances. Rush jobs are never good. But the expense of down-time may be so large that they are necessary. I have worked on process control applications where you sometimes have to put fixes in with a bare minimum of testing because the expense of having the plant idle is enormous; this might also apply to very big websites. I was rather nervous about this, but it generally worked, or the resulting system was good enough to be going on with. Full testing could wait until the plant was back on again.

So the answer is; none. All the 'procedure' could wait until after the fix is applied.

Brian Hooper
+1  A: 

Testing the fix in a beta environment is helpful.

jdot
Definitely. I wouldn't put anything in production without testing it once.
C. Ross
+3  A: 

Short answer: you don't. But you can mitigate some of the issues.

Even for my own, open-source code, I have a PC sitting here that does nothing but integration testing (unit testing is fine, but is not the panacea many think it is; it's quite possible to say both "all my tests pass!" and "my program doesn't work!"). For web stuff it's got Selenium installed, and exhaustively tests the code to make sure it works sanely. If I have to rush a patch on something, I can at least wait for Selenium to return before pushing it; almost nothing is that time-critical (and if it is, boy did you screw up and it's time to hash out a fast damage-control patch and then go back and do it right). The one caveat is that Selenium and friends (including desktop applications) cannot perform meaningful system testing unless they are on identical hardware to the Real Thing. (For desktop applications, especially consumer-released ones, this kind of system testing is nearly impossible, but given enough money you can build a reasonably broad testing matrix.)

Unit testing is nice and good for fast turnarounds, but I find that you find the really nasty problems in your integration tests.

Ed Ropple
+14  A: 
  1. Minimise the changes. Don't fix the architecture "while you're in there". Don't refactor. Make the fix at the level which requires the least number of interface changes to accommodate it, even if it's not really the proper level of responsibility, or if you've broken some encapsulation somewhere. If the minimal fix differs from the best fix, then I refer to the minimal fix as the "bodge" and the best fix as the "proper fix".
  2. Having bodged it, run all the tests you have, including the ones you don't normally bother with. Remember that verification will be under time pressure too, so just like you they're more likely to make mistakes than usual.
  3. Having sent the bodge for verification, consider what you've done in light of what the proper fix would be, and if necessary write new tests relating to the area(s) you've changed. If you're lucky, all your new tests will pass. If you're not lucky, and you've introduced new bugs, it's still better to know about them than not.
  4. Go back and make the fix "properly", run all the tests again, and compare behaviour between the original broken code, the bodge, and the proper fix. If the bodge is as good as the proper fix from the POV of behaviour, then it's probably safe to ship the bodge (which with luck is already way down the release pipeline) now, and switch to the more elegant, rigorous, and maintainable proper fix via the normal release cycle.
  5. It's surprising how often "duress" actually means "hurry up and wait". A sensible analysis of the release process suggests that your code needs to be done today, hence your tight deadline. If missed, multiple people will have aneurysms through stress. In fact, you deliver that, but by the end of the day after tomorrow when the proper fix is done, your rush-job hasn't been used yet. So you replace it with the proper fix.
Steve Jessop
A: 

Lots and lots of caffeine.

dsimcha
+5  A: 

Couple of suggestions:

  1. Make sure hotfixes go on a production branch, the main development trunk should not have a bad patch.
  2. If time is not on your side, make the hotfix but use your bug-tracking system to get back to this issue once the release is out. Put the proper fix in all branches.
  3. Under all circumstances have a regression that tests the product out -- hotfix should not introduce the need for more hotfixes or situation will get real hot.
  4. Maintain unit testing framework for individual source snippets -- in particular the complex functions and classes. Run them before you put in your hotfix. Look into cppUnit, cppTest or boost test frameworks for this.
  5. For C/C++ code have a valgrind or purify run -- at least on a subset of tests before you make a release.
  6. Look into related problem scenarios. I know this one's tough to do under stress, but if you possibly can engage someone else to do some Q&A before releasing.
  7. Pair programming, code review
  8. Practice deep breathing and don't consume much caffeine
Fanatic23
Definitely on the caffeine, though I might say "only as much caffienne as necessary".
C. Ross
"hotfix should not introduce the need for more hotfixes or situation will get real hot" the technical term is "critical chain reaction" ;-)
Steve Jessop
A: 

I always start by outlining exactly which problems are immediate fixes and which are just updates or improvements. Then, I ignore the updates and do only the fixes to reduce the size of the task. Then, I create a rough outline of the process that needs to happen. Something like: 1. user enters info, 2, form validates, 3. if validation error.... It is not an entire logic flow chart, since there is not enough time. But having ANYTHING laid out in writing REALLY helps me keep on track. Planning is essential... how much planning is subjective. I try to do only as much as the time I have will allow to do the most effective job. I am ALWAYS under deadlines and clients rarely understand that their "simple" changes are rarely simple.

exoboy
+1  A: 

This is on the touchy-feely side, but one thing to mitigate the danger is to take your proposed fix to another (senior) engineer on your team and say "I think this is the right thing for tonight; I see risks A, B, and C; I considered and rejected solutions X and Y." Once implemented and tested, go back to that engineer and have them review the change with you line-by-line.

If your fix is the Right Thing in the situation, those steps will add less than half an hour to the timeline. If your buddy points out the catastrophic lossage that will occur if you do it, the time it takes to work out a better fix is well worth it.

Russell Borogove
+1, although I would note that even when not under duress, a second opinion tends to improve the quality of my decisions :-)
Steve Jessop
Yes, I recommend checkin buddies for all changes, not just hasty ones.
Russell Borogove