views:

101

answers:

3

Hi there, I have a calculation that generates what appears to be the Float 22.23, and a literal 22.23 like so:

some_object.total => 22.23
some_object.total.class => Float

22.23 => 22.23
22.23.class => Float

But for some reason, the following is false:

some_object.total == 22.23 ? true : false

Wacky, right?

Is there some kind of precision mechanism being used that maybe isn't completely transparent through the some_object.total call?

+1  A: 

there is something else going on here. this is from a 1.8.7 irb

irb(main):001:0> class Test
irb(main):002:1> attr_accessor :thing
irb(main):003:1> end
=> nil
irb(main):004:0> t = Test.new
=> #<Test:0x480ab78>
irb(main):005:0> t.thing = 22.5
=> 22.5
irb(main):006:0> t.thing == 22.5
=> true
Matt Briggs
the total is actually a sum and calculation of several other numbers, causing the inaccuracies.
btelles
+5  A: 

Float#to_s and Float#inspect round. Try "%.30f" % some_object.total and you will see that it's not quite 22.23.

sepp2k
heh..."22.229999999999996873611962655559" That's it. Thanks.
btelles
+3  A: 

Floating-point numbers cannot precisely represent all decimal numbers within their range. For example, 0.9 is not exactly 0.9, it's a number really close to 0.9 that winds up being printed as it in most cases. As you do floating-point calculations, these errors can accumulate and you wind up with something very close to the right number but not exactly equal to it. For example, 0.3 * 3 == 0.9 will return false. This is the case in every computer language you will ever use — it's just how binary floating-point math works. See, for example, this question about Haskell.

To test for floating point equality, you generally want to test whether the number is within some tiny range of the target. So, for example:

def float_equal(a, b)
  if a + 0.00001 > b and a - 0.00001 < b
    true
  else
    false
  end
end

You can also use the BigDecimal class in Ruby to represent arbitrary decimal numbers.

If this is a test case, you can use assert_in_delta:

def test_some_object_total_is_calculated_correctly
  assert_in_delta 22.23, some_object.total, 0.01
end
Chuck
Yup, that helps! Thanks!
btelles
Just to make obvious if it is not from Chuck's post, this is not a ruby floating point limitation. This is a floating point limitation, period. It works this way in every language that uses IEEE floating point. Numerical issues are an important and somewhat deep topic that you would cover at some point in any computer science curriculum.
frankc