Floating point accuracy in F# (and .NET)

Question

1. In "F# for Scientists" Jon Harrop says:

   Roughly speaking, values of type int approximate real numbers between min-int and max-int with a constant absolute error of +- 1/2 whereas values of the type float have an approximately-constant relative error that is a tiny fraction of a percent.

   Now, what does it mean? Int type is inaccurate?

2. Why C# for (1 - 0.9) returns 0.1 but F# returns 0.099999999999978 ? Is C# more accurate and suitable for scientific calculations?

3. Should we use decimal values instead of double/float for scientific calculations?

Answer 1

No, F# and C# uses the same double type. Floating point is almost always inexact. Integers are exact though.

UPDATE:

The reason why you are seeing a difference is due to the printing of the number, not the actual representation.

Answer 2

For the first point, I'd say it says that int can be used to represent any real number in the intger's range, with a constant maximum error in [-0,5, 0.5]. This makes sense. For instance, pi could be represented by the integer value 3, with an error smaller than 0.15.

Floating point numbers don't share this property; their maximum absolute error is not independent of the value you're trying to represent.

Answer 3

3 - This depends on calculations: sometimes float is a good choice, sometimes you can use int. But there are tasks when you lack of precision for any of float and decimal.

The reason against using int:

> 1/2;; 
val it : int = 0

The reason against using float (also known as double in C#):

> (1E-10 + 1E+10) - 1E+10;;
val it : float = 0.0

The reason against BCL decimal:

> decimal 1E-100;;
val it : decimal = 0M

Every listed type has it's own drawbacks.

Answer 4

1. For an arbitrary real number, either an integral type or a floating point type is only going to provide an approximation. The integral approximation will never be off by more than 0.5 in one direction or the other (assuming that the real number fits within the range of that integral type). The floating point approximation will never be off by more than a small percentage (again, assuming that the real is within the range of values supported by that floating point type). This means that for smaller values, floating point types will provide closer approximations (e.g. storing an approximation to PI in a float is going to be much more accurate than the int approximation 3). However, for very large values, the integral type's approximation will actually be better than the floating point type's (e.g. consider the value 9223372036854775806.7, which is only off by 0.3 when represented as 9223372036854775807 as a long, but which is represented by 9223372036854780000.000000 when stored as a float).

2. This is just an artifact of how you're printing the values out. 9/10 and 1/10 cannot be exactly represented as floating point values (because the denominator isn't a power of two), just as 1/3 can't be exactly written as a decimal (you get 0.333... where the 3's repeat forever). Regardless of the .NET language you use, the internal representation of this value is going to be the same, but different ways of printing the value may display it differently. Note that if you evaluate 1.0 - 0.9 in FSI, the result is displayed as 0.1 (at least on my computer).

3. What type you use in scientific calculations will depend on exactly what you're trying to achieve. Your answer is generally only going to be approximately accurate. How accurate do you need it to be? What are your performance requirements? I believe that the decimal type is actually a fixed point number, which may make it inappropriate for calculations involving very small or very large values. Note also that F# includes arbitrary precision rational numbers (with the BigNum type), which may also be appropriate depending on your input.