views:

1249

answers:

2

When 2 decimal(30,10) numbers are divided in Sql Server 05, 2 last decimals seem to be getting lost (not even rounded off, simply truncated).

For example:

Declare @x decimal(30,10)
Declare @y decimal(30,10)
Declare @z decimal(30,10)

select @x = 2.1277164747 
select @y = 4.8553794574

Select @z = @y/@x   
select @z

Result: 2.28196731**00**

But if 2 numbers being divided are converted to float that seems to work:

....
Select @z = cast(@y as float)/cast(@x as float)
select @z

Result: 2.28196731**81**

Why is Sql doing this? And what's the right way of dividing decimals without loosing precision in Sql.

A: 

In short, use casting to guarantee your results. When you assign @x and @y to literal values, they are probably adopting the precision of those literals. This helps to explain why division by those values comes up short of your expectations.

David Andres
if you run _SELECT @x,@y_ you will see that @x and @y contain all the decimal digits
KM
The literal values of @x and @y are stored properly, it's the result stored in @z that is being truncated.
richardtallent
@All..thanks for the clarifications.
David Andres
+10  A: 

The maximum precision allowed in SQL Server is 38. You are using Decimal(30,10). The max value is 99,999,999,999,999,999,999.9999999999 if you divide this number by 0.000000001, you will end up with an even bigger number, so the resulting data type must be able to accommodate it. This causes you to lose some precision.

Change your original data types to Decimal(20,10) and this problem does not occur.

For full rules regarding data types (and how they are affected by math operations):

Full rules here

G Mastros
This is the correct answer. SQL Server determines a precision and scale for the result based on the precision and scale of the inputs *types*, not the input *values*. So, the result is calculated as decimal(30,8) and then converted to decimal(30,10) to store in @z.
richardtallent
This makes sense, but>>This causes you to lose some precision<<. It wouldn't be so bad if Sql were to round off to 8 decimals, but it simply chops off the last 2 decimals causing erroneous results. It should at least give some kind of warning, or even through an error when this happens
WebMatrix
One thing people seem to miss in particular is that the scale is reduiced to 6 digits when doing division on 2 maximum-precision numbers. Decimal(38,10) / decimal(38,10) = decimal (38,6). I ran down an explanation at one point that they have to do this in to protect the precision in the internal workings in case of overflow.
DaveE
@richardtallent, the resulting precision is actually decimal(38,8). Add this to the code above: Select SQL_VARIANT_PROPERTY(@y/@x, 'Precision'), SQL_VARIANT_PROPERTY(@y/@x, 'Scale')
G Mastros
I blogged about this about a year ago:http://blogs.lessthandot.com/index.php/DataMgmt/DataDesign/sql-server-precision-and-scale-problems
G Mastros
@G Mastros: doh! yes, of course... ugh, can't edit comments unfortunately.@WebMatrix: remember, CPUs don't think in decimals. Internally, SQL Server guesses the output precision/scale needed and performs an integer operation. It makes the assumption that places before the decimal point should be preserved rather than those afterward.
richardtallent