views:

86

answers:

4

Can anyone explain this weirdness:

Dim result as single = 0
result = CType("8.01", Single) * 100 ' result=801.0 as expected
result = CType("8.02", Single) * 100 ' result=802.000061 --- not expected

Further to the above

result = 8.02 * 100 ' result = 802.0 as expected
+2  A: 

Single (or float) only has seven significant digits, so everything that might be printed beyond those is pretty much bogus anyway. This is a normal artifact of floating-point arithmetic. You can only rely on up to 4 digits after the decimal point with your number (as three significant digits are already before the decimal point).

Further elaborated, the numbers 8.01 and 8.02 can't be represented exactly using binary storage (as neither 0.01 nor 0.02 can be represented exactly with sums of fractions of the form 1/2n). The details may vary from number to number but you may see extraneous digits outside the normal precision range when using such numbers.

This doesn't affect the fact that 801 and 802 can be represented exactly but you have no exact number to start with in this case.

ETA: In fact, what you see when you include the calculation directly is just this: The compiler will evaluate the calculation for you and simply write 802 into the program. You can use Reflector to verify that. Also floating point literals in the source code are likely Double by default so you have much greater precision to begin with here. If result is a Single this will be downcasted to Single and the error in the 16th digit after the decimal point is simply thrown away since it can't fit into Single anyway.

Joey
Thanks Johannes,You would have replied to my answer before I modified the post to add the Further section. What you say makes sense.
Andrew
+1  A: 

This is due to a limitation in the way floating point numbers are represented in memory. Have a read through the wikipedia page for the full lowdown - it's not an error, or something specific to .net.

EDIT: these two articles are worth a look too (the latter is math heavy).

rob
A: 

If what you need is to compare floating point values, see here: http://stackoverflow.com/questions/1530069/comparing-floating-point-values/1530166#1530166

Konamiman
A: 

Single and Double in .NET are represented as floating point numbers and unfortunately storing these kinds of numbers is not precise. You will get small discrepancies when working with 'Single' and 'Double' numbers.

However there is an alternative - .NET has a Decimal type which is much better at dealing with these numbers. It's still not perfect (see http://www.yoda.arachsys.com/csharp/decimal.html - in particular the last paragraph) but will be much better at representing quantities like money.

Richard Downer