views:

271

answers:

5

This is very strange to me:

irb(main):012:0> "100.7".to_f.modulo(1)
=> 0.700000000000003

Why the 3 at the end?

irb(main):019:0> "10.7".to_f.modulo(1)
=> 0.699999999999999

Same thing here...we are only getting the remainder of this value divided by one. It should be exact.

A: 

this is because it's not possible to represent all floating point numbers exactly.

Peter
+3  A: 

This is typical floating point rounding. You simply cannot express every single decimal number in the fixed number of bits in a Float, so some values are rounded to the nearest value that can be represented.

Due to this, it is suggested that you don't compare Floats for equality. Compare for less than or greater than, but never exact equality.

http://en.wikipedia.org/wiki/Floating%5Fpoint#Representable%5Fnumbers.2C%5Fconversion%5Fand%5Frounding

Simply, it is not the case that "it should be exact". Don't expect that from floating point decimals.

Joe Koberg
+13  A: 

What Every Computer Scientist Should Know About Floating-Point Arithmetic

sepp2k
Or, in brief, floating point is evil.
Steven Sudit
+1  A: 

Floating points are not exact. The short version is it's not possible to store an infinite amount of values in a finite amount of bits.

The longer version is What Every Computer Scientist Should Know About Floating-Point Arithmetic

leeeroy
+2  A: 

Welcome to floating point math. There are many numbers which cannot be represented in standard floating point notation and come out just a tiny bit off.

This is easily illustrated as follows:

(1..10).collect do |i|
  v = ((10**i).to_f + 0.7)
  puts "%13.1f = %.30f" % [ v, v.modulo(1) ]
end

Where the result is:

         10.7 = 0.699999999999999289457264239900
        100.7 = 0.700000000000002842170943040401
       1000.7 = 0.700000000000045474735088646412
      10000.7 = 0.700000000000727595761418342590
     100000.7 = 0.699999999997089616954326629639
    1000000.7 = 0.699999999953433871269226074219
   10000000.7 = 0.699999999254941940307617187500
  100000000.7 = 0.700000002980232238769531250000
 1000000000.7 = 0.700000047683715820312500000000
10000000000.7 = 0.700000762939453125000000000000

Notice that the larger the number gets, the lower the precision beyond the decimal place. This is because there is a fixed amount of precision available to represent the entire number.

tadman