tags:

views:

276

answers:

5

I'm using decimal type (.net), and I want to see if two numbers are equal. But I only want to be accurate to 10 decimal places. For example take these three numbers. I want them all to be equal.

0.1123456789
0.11234567891
0.11234567899

The 10 decimal limit is coming from my database, so I have to assume that the first number was already rounded, and therefore I can't simply round the others because the last one will round up.

I really just want to truncate the to 10 decimal places, but can't see how to do this either.

+5  A: 

What about multiplying by 10^10 and dropping the fractional part?

decimal x2 = Math.Truncate(x * 10000000000);
decimal y2 = Math.Truncate(y * 10000000000);
Assert.Equals(x2, y2);

EDIT: Changed to Math.Truncate by Aaron's suggestion. Thanks.

Martinho Fernandes
Of course this will also be wrong for any value greater than 2^31 / 10000000000.
Aaron
@Aaron, definitely.
Martinho Fernandes
Change it to Math.Truncate() instead of converting to int - you'll still lose a little top-end precision but it will be 'closer'.
Aaron
@Aaron - The example the OP showed would be for numbers less than 1, so this would work fine.
James Black
@James: yeah, but there's nothing wrong with improvement.
Martinho Fernandes
The DB field I'm trying to match is Decimal(28,10), so I'd like the test to be valid for all numbers that could be stored in the DB. But generally, I'm dealing with percentages so they are stored as 0-100. Thanks.
TheSean
+1  A: 

Multiply by 10000000000, convert to an int, then divide by the same number.

That way you truncate the the excess digits.

You may want to add .5 * 10^-11 in order to round properly before truncating.

James Black
A: 

Maybe this works for you:

If Decimal.Round(yourDec1, 10, MidpointRounding.AwayFromZero) = Decimal.Round(yourDec2, 10, MidpointRounding.AwayFromZero) Then

Bobby

Bobby
Rounding != Truncating.
Martinho Fernandes
+6  A: 

The same way you'd compare floating point numbers. Here's some pseudocode, because I don't know the .NET call for absolute value, but it will essentially look like this (modify the constant for the precision needed):

if( Math.Abs( value1 - value2 ) < 0.0000000001 )
{
  // blah blah
}
Russell Newquist
return (Math.Abs(value1 - value2) <= precision);
Richard Morgan
@Russel, fixed to use `Math.Abs`.
Martinho Fernandes
Thanks for the assist. :D
Russell Newquist
Clever, I like it. In addition to testing equality, I also need to test greaterthan/lessthan, so this won't work for that.
TheSean
A: 

Multiply by 10^10 Rather than converting to an integer (which is 32 bit) it might be worth using an int64. An int 32 has a limit of approximatly 2 billion which when multipled out gives a range between +2.1 and -2.1 on your decimal value, an int64 has a much larger range that, when multiplied out gives you a range of approximatly +922,000,000 to -922,000,000 on the decimal value

http://msdn.microsoft.com/en-us/library/system.int32.aspx

http://msdn.microsoft.com/en-us/library/system.int64.aspx

zeocrash