tags:

views:

106

answers:

3

Does Ruby have a bug in its rounding? Why does it behave like this:

>> [1.14, 1.15, 1.16].map{|x| "%.1f" % x}
=> ["1.1", "1.1", "1.2"]
>> [1.4, 1.5, 1.6].map{|x| "%.0f" % x}
=> ["1", "2", "2"]

as in, why does 1.15 get rounded to 1.1, but 1.5 gets rounded to 2? At the very least, isn't this inconsistent? the behaviour is the same in ruby 1.9.1 and ruby 1.8.7.

+2  A: 

You're using floating point numbers. Floating point numbers aren't precise. See http://en.wikipedia.org/wiki/IEEE_754-2008 for an introduction in the standard.

The short version is: NEVER use floats for anything where you need precision in any way!

johannes
+4  A: 

Take a look at my answer to this question

http://stackoverflow.com/questions/1651671/why-does-perls-sprintf-not-round-floating-point-numbers-correctly/1651731#1651731

This may be the same thing

Jim Garrison
heh heh, I didn't even think of this. makes much sense in retrospect.
Peter
+1  A: 

It's useful to recall and also quite ironic to contemplate, but floating point numbers only represent exactly: (a) a few fractions or (b) all integers.

So, to have an exact representation a fraction must be composed of (negative) powers of two. So, the following fractions are the only ones between .01 and .99 that are exactly represented:

0.25
0.50
0.75

In other words, FP is perfectly accurate when dealing with integers. Go figure.

DigitalRoss
That doesn't sound right. You should get an exact answer for all the negative exponents that can be represented in the mantissa (which Wikipedia tells me is now called the "significand" -- must have happened after I went to school). So `2**1/n` should work for any `n` that can fit in the mantissa. No?
zetetic
should read "for any result that can fit in the mantissa"
zetetic
Almost, but only when *n* is a power of two. So, a fraction composed of one or more terms from *1/2, 1/4, 1/8, 1/16, 1/32*, ... can be represented exactly.
DigitalRoss
"when n is a power of two" -- uh, right, that is what I meant to say :)
zetetic