views:

204

answers:

2

I have come across a precision issue with double in .NET I thought this only applied to floats but now I see that double is a float.

double test = 278.97 - 90.46;
Debug.WriteLine(test) //188.51000000000005

//correct answer is 188.51

What is the correct way to handle this? Round? Lop off the unneeded decimal places?

+4  A: 

Use the decimal data type. "The Decimal value type is appropriate for financial calculations requiring large numbers of significant integral and fractional digits and no round-off errors."

Michael Petito
+1 If you're doing financial stuff, you *really* need to be using `decimal`.
Dean Harding
Decimal is the correct data type you should be using for financial calculations. It does not eliminate the need for rounding but minimizes errors due to rounding.
Scott Dorman
Thanks. This fixed my issue.
Alistair
+1  A: 

This happens in many languages and stems from the fact that we cannot usually store doubles or floats precisely in digital hardware. For a detailed explanation of how doubles, floats, and all other floating point values are often stored, look at the various IEEE specs described on wikipedia. For example: http://en.wikipedia.org/wiki/Double_precision

Of course there are other formats, such as fixed-point format. But this imprecision is in most languages, and why you often need to use epsilon tests instead of equality tests when using doubles in conditionals (i.e. abs(x - y) <= 0.0001 instead of x == y).

How you deal with this imprecision is up to you, and depends on your application.

foxwoods