How much a performance boost is worthy/notable?

Question

Everyone has their views on when and where to increase performance, so that's not of interest here, but I was wondering at what point a performance gain becomes worthy of promotion (say from branch development into the trunk, or enough to be placed on some progress report or whatever).

For example my rule-of-thumb is a 10% performance boost is noteworthy, but a 5% is nothing special (of course added up they may be worthy though)

Note: as there is no correct answer marking this wiki, but I do feel having opinions on the subject is useful when making decision about performance

Answer 1

O(n) to O(log n) algorithm improvement for big n's is very good.

Answer 2

I think it depends on the elegance/readability of the code too. If a particular change feels like a more correct solution, eliminating some stupidity in the code then if it also improves performance at all, it's "worthy". However, if a particular change is designed purely for performance, and decreases code readability or maintainability, then it better be a pretty good boost to make it worthy... maybe 50%.

Answer 3

I'd rather use absolute values. 20% of 1 ms isn't something you care about. But 2% of an hour is pretty impressive.

Answer 4

If increased performance adds a new quality to the system, it's worth mentioning. Otherwise noone usually cares.

For example, if the system used to work noticeably slow when loaded with a reasonable but above usual amount of data and you've fixed this problem it's worth mentioning because those users who load huge amounts of data will benefit from the improvement.

Answer 5

It depends.

If those 5% translate into money saved e.g. on a big server farm it would be noteworthy.

If it is 5% faster on an end user's computer, who wouldn't even notice, then not.

Answer 6

It depends on context.

In an application that needs to play video in real-time, a 1ms improvement per frame can be the difference between 30 frames per second (which is a common rate for video clips) to 29 frames per second (which would look - visually - much worse due to dropping frames).

In another application that runs over-night, two hours vs. three hours may not make any real world difference (but the load caused to the database while running can be very significant - optimize that instead!).

Answer 7

Basically when it is an order of magnitude or better for a complete unit of work.

• Hours to minutes batch processes
• Minutes to seconds user triggered batch processes, installs, builds
• Seconds to a second user triggered processes
• A second to instant user triggered processes

Answer 8

Here's a simple test: give someone the unoptimized system and ask them to use it. Give them the optimized system and ask them to use it. Ask them if they can tell the difference, and which they like best. (be sure that it's a double-blind test and that the systems are otherwise equal). If people like the optimized system more, it's worth making noise about.

Here's a simpler test: can you tell the difference? If you can't, then it's not worth mentioning. (on the other hand, small improvements accumulate).