views:

575

answers:

2

I did some testing with floating point calculations to minimize the precision loss. I stumbled across a phenomen I want to show here and hopefully get an explanation.

When I write

print 1.0 / (1.0 / 60.0)

the result is

60.0024000960

When I write the same formula and do explicit casting to float

print cast(1.0 as float) / (cast(1.0 as float) / cast(60.0 as float))

the result is

60


Until now I thought that numeric literals with decimal places are automatically treated as float values with the appropriate precision. Casting to real shows the same result as casting to float.

  • Is there some documentation on how SQL Server evaluates numeric literals?
  • Of what datatype are those literals?
  • Do I really have to cast them to float get better precision (which sounds like irony to me :)?
  • Is there an easier way than cluttering my formulas with casts?
+4  A: 

SQL Server uses the smallest possible datatype.

When you run this script

SELECT SQL_VARIANT_PROPERTY(1.0, 'BaseType')
SELECT SQL_VARIANT_PROPERTY(1.0, 'Precision')
SELECT SQL_VARIANT_PROPERTY(1.0, 'Scale')
SELECT SQL_VARIANT_PROPERTY(1.0, 'TotalBytes')

you'll see that SQL Server implicitly used a NUMERIC(2, 1) datatype.
The division by 60.0 converts the result to NUMERIC(8, 6).
The final calculation converts the result to NUMERIC(17, 10).


Edit

Taken from SQL Server Books Online Data Type Conversion

In Transact-SQL statements, a constant with a decimal point is automatically converted into a numeric data value, using the minimum precision and scale necessary. For example, the constant 12.345 is converted into a numeric value with a precision of 5 and a scale of 3.

Lieven
Ah.. I didn't know that method until now. Thank you very much :)
VVS
So there's no way around an explicit cast to get the expected result? Is there another way to tell SQL Server that a given literal is of type (e.g.) float?
VVS
@VVS, besides an explicit cast as you already are doing, not that I know of. SQL Server always interpretes the value as numeric. The problem can be alleviated a bit if you yourself add precision to your values like 1.000000000000000000000000000000000000.
Lieven
@Lieven: do you have any kind of documentation that confirms your statement that "SQL Server always interpretes the value as numeric"?
VVS
@VSS: I have added a reference to the documentation.
Lieven
+1  A: 

Yes, you frequently have to cast them to float get better precision. My take on it:

For better precision cast decimals before calculations

AlexKuznetsov