views:

97

answers:

1

I have searched and can not find the answer. I double checked the data types between SQL and CLR and I appear to have that correct. But I am getting a different result between using CLR and SQL. Not much, but enough to be off a penny. And that is not acceptable.

Example in VB.NET

Dim dLoanAmount As Decimal = 169500  
Dim dNetDiscount As Decimal = 100.871  
Dim dDiscountPremium As Decimal = (dLoanAmount * (dNetDiscount - 100.0) / 100.0) 
Console.WriteLine("original amount is " + dDiscountPremium.ToString())

will display 1476.34499999999

Example in SQL

DECLARE @loanAmt decimal (20,10)  
DECLARE @discount decimal (20,10)  
SET @loanAmt = 169500.000000  
SET @discount = 100.871000  
select   @loanAmt*(@discount-100.0)/100.0

that returns 1476.345000000000000

We have to use the VB for some documents, but for some file transfers we use sql. Anyone have any suggestions why this is?

cheers
bob

+1  A: 

You're using double literals instead of decimal ones. Try this:

Dim dLoanAmount As Decimal = 169500D
Dim dNetDiscount As Decimal = 100.871D  
Dim dDiscountPremium As Decimal = (dLoanAmount * (dNetDiscount - 100D) / 100D) 
Console.WriteLine("original amount is " + dDiscountPremium.ToString())
Jon Skeet
First, welcome back
astander
That worked out great, but I am pulling the data from a database and those were the values. I just set them for the example. The data type in the table is decimal(16,6) for both the loan amount and discount. Thanks for your time.
Bob Cummings
You need to find out *exactly* what the values are in both the database and .NET then. As you can see, when you get the right values into .NET, the arithmetic works fine.
Jon Skeet
Jon I sure appreciate your time. Unfortunately those are the exact values in the database. That is why I am confused. And the data types match up SqlDecimal to CLR Decimal according to the MSDN documentation.
Bob Cummings
@Bob: They clearly aren't the exact values by the time you've extracted them from the database - because as my example shows, if you *do* give .NET the right values, you get back the right result. Try to work out *exactly* what value you're getting in .NET - if you could show how you're fetching the values, that would help too. You can print out a "round trip" value with the "r" format specifier: `Console.WriteLine(value.ToString("r"))`
Jon Skeet
@Jon I found the problem. For some reason when dNetDiscount - 100D it returns 0.870999999999995 instead of .871. If I instead do (dNetDiscount * 1000 - 100000) / 1000) I get the expected result .871 I think it is because I am moving it *up* to an integer value, doing my calculation, and then moving it back *down* to a decimal value.
Bob Cummings