T-SQL Decimal Division Accuracy

Question

Hi,

Does anyone know why, using SQLServer 2005

SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,9),12499999.9999)

gives me 11.74438969709659,

but when I increase the decimal places on the denominator to 15, I get a less accurate answer:

SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,15),12499999.9999)

give me 11.74438969

Answer 1

BTW The windows calculator gives 11.744389697096595117576772760941

Answer 2

Convert the expression not the arguments.

select CONVERT(DECIMAL(38,36),146804871.212533 / 12499999.9999)

Answer 3

For multiplication we simply add the number of decimal places in each argument together (using pen and paper) to work out output dec places.

But division just blows your head apart. I'm off to lie down now.

In SQL terms though, it's exactly as expected.

--Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
--Scale = max(6, s1 + p2 + 1)

--Scale = 15 + 38 + 1 = 54
--Precision = 30 - 15 + 9 + 54 = 72
--Max P = 38, P & S are linked, so (72,54) -> (38,29)
--So, we have 38,20 output (but we don use 20 d.p. for this sum) = 11.74438969709659
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,9),12499999.9999)


--Scale = 15 + 38 + 1 = 54
--Precision = 30 - 15 + 15 + 54 = 84
--Max P = 38, P & S are linked, so (84,54) -> (38,8)
--So, we have 38,8 output = 11.74438969
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,15),12499999.9999)

You can do the same math if follow this rule too, if you treat each number pair as

• 146804871.212533000000000 and 12499999.999900000
• 146804871.212533000000000 and 12499999.999900000000000

Answer 4

To put it shortly, use DECIMAL(25,13) and you'll be fine with all calculations - you'll get precision right as declared: 12 digits before decimal dot, and 13 decimal digits after. Rule is: p+s must equal 38 and you will be on safe side! Why is this? Because of very bad implementation of arithmetic in SQL Server! Until they fix it, follow that rule.

Answer 5

i've noticed if you cast the dividing value to float gives you the correct answer.

i.e

select 49/30 (result = 1)

would be

select 49/cast(30 as float) (result = 1.63333333333333)

Answer 6

We were puzzling over the magic transition,

P & S are linked, so:

1. (72,54) -> (38,29)

2. (84,54) -> (38,8)

Assuming (38,29) is a typo and should be (38,20), the following is the math:

1. i. 72 - 38 = 34, ii. 54 - 34 = 20

2. i. 84 - 58 = 46, ii. 54 - 46 = 8

And this is the reasoning:

i. Output precision less max precision is the digits we're going to throw away.

ii. Then output scale less what we're going to throw away gives us... remaining digits in the output scale.

Hope this helps anyone else trying to make sense of this.