F#+.Net, calculation error using the System.Math.Floor function

Question

Hi everybody, here's a function I wrote to print every digit of a float number in F#:

let rec TestFloor (fnum:float) =
    let floor = System.Math.Floor(fnum)
    printfn "fnum:%f floor:%f" fnum floor
    if floor > 0.0 then TestFloor((fnum - floor) * 10.0)

Anyway the result is strange, for example:

> TestFloor 1.23;;
fnum:1.230000 floor:1.000000
fnum:2.300000 floor:2.000000
**fnum:3.000000 floor:2.000000**
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:10.000000 floor:9.000000
fnum:9.999998 floor:9.000000
fnum:9.999982 floor:9.000000
fnum:9.999822 floor:9.000000
...

At the fourth line the floor of 3.0 turned out to be 2.0 which is strange. The following computations are all going wrong. I'm wondering what's happening there?

Thanks!

---

Edit 2

@ sepp2k

This is the result after using %.30f:

> TestFloor 1.23;;
fnum:1.230000000000000000000000000000 floor:1.000000000000000000000000000000
fnum:2.300000000000000000000000000000 floor:2.000000000000000000000000000000
**fnum:3.000000000000000000000000000000 floor:2.000000000000000000000000000000**
fnum:9.999999999999980000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999999820000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999998220000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999982240000000000000000 floor:9.000000000000000000000000000000
fnum:9.999999999822360000000000000000 floor:9.000000000000000000000000000000

As you can see at the fourth line fnum is 3.00... and the floor value is 2.00..

---

Edit 3 - Solved

Thank you all, now I understand what the problem is about.

Answer 1

Presumably the value of fnum is something like 2.9999999999999999999 and printfn just rounds that to 3.0 when displaying it. Try to increase the number of displayed digits with something like %.30f instead of %f.

Answer 2

This is just one of those rounding issues you get with floating point arithmetic. You will find its representation is 2.99999... probably infinitely recurring. The call to printfn clearly rounds this up to the expected representation, but floor still sees this as 2.99999...

This is exactly the sort of problem that the decimal type exists to solve, so if we rewrite to use decimal, we obtain the correct result:

let rec TestFloor dnum =
    let fl = floor dnum
    printfn "fnum:%f floor:%f" dnum fl
    if fl > 0.0M then TestFloor((dnum - fl) * 10.0M)

This gives:

> TestFloor 1.23M;;
fnum:1.230000 floor:1.000000
fnum:2.300000 floor:2.000000
fnum:3.000000 floor:3.000000
fnum:0.000000 floor:0.000000
val it : unit = ()

Of course, you can stick with float, but add a very small tolerance value to ensure that these corner cases are always just a little above the expected value, rather than just a little below, eg:

let rec TestFloor fnum =
    let fl = floor (fnum + 0.00000000001)
    printfn "fnum:%f floor:%f" fnum fl
    if fl > 0.0 then TestFloor((fnum - fl) * 10.0)

Which gives the same result as above.

Answer 3

Even though you get 3.00000... at max decimal precision, there might still be a rounding. Let's not forget the value is stored in binary, and a rounding is always needed for printing purposes (binary to decimal). The answer is, although your value prints as 3.0000..., it is actually less than 3. Adding more decimal digits won't cut it.

Consider this code:

> let num = 3.0 - 2.**(-51.);;
> printfn "%.30f" num;;
3.000000000000000000000000000000
> floor num;;
val it : float = 2.0