Is casting narrow types to wider types to save memory and keep high-precision calculations a terrible idea?

Question

I'm dealing with financial data, so there's a lot of it and it needs to be relatively high-precision (64bit floating point or wider).

The standard practice around my workplace seems to be to represent all of it as the c# decimal type which is a 128bit wide floating point specifically created to support round-off free base10 operations.

Since 64bit is wide enough to maintain the representative precision, is it ridiculous to cast the data to the wider type for all calculations (mult,div,add,etc) and then back to 64bit for sitting in memory (which is where it spends of most if its time)?

For reference: memory is definitely the limiting resource here.

Answer 1

This seems perfectly sane, if 64 bit floating point is truly enough to represent the precision you want. The extra precision decimal is, as you say, often used purely to minimize cumulative errors over multiple operations.

Answer 2

You should use a profiler to see what objects are taking up a lot of memory. If your decimal objects are the culprit, then I would say yes go after them. Otherwise you are just making guesses. Profiler will tell you for sure.

Answer 3

You need to do the numerical analysis to see if the practice (of keeping 128 bits) is ridiculous, or just lazy, or really necessary.

Answer 4

It is perfectly reasonable to store your numbers at 64 bit, cast them to the decimal type for calculations, and cast the result back to 64 bit, if you don't mind the performance hit.

We require this level of precision where I work, so this is exactly what we do here. We take a two orders of magnitude hit in speed by doing the cast, but we never have to worry about large errors in the floating point arithmetic. Without the cast, the calculation can be wildly inaccurate, depending on the range of the numbers and the type of calculation being performed.

For more on floating point arithmetic, and why errors can creep into your calculations, see "What Every Computer Scientist Should Know About Floating-Point Arithmetic" at http://docs.sun.com/source/806-3568/ncg_goldberg.html

Answer 5

The point of using decimal (128 bits) over double (64 bits) and float (32 bits) isn't usually to do with the size. It's to do with the base. While double and float are floating binary point types, decimal is a floating decimal point type - and it's that feature that lets it represent numbers like 0.1 exactly where float/double can't.

There's no conceptual reason why we couldn't haven't a 64-bit decimal type, and in many cases that would indeed be enough - but until such a type comes along or you write it yourself, please don't use the "shorter" (and binary floating point) types of float/double for financial calculations. If you do, you're asking for trouble.

If you're suggesting writing a storage type which can convert to/from decimal and is still a floating decimal type, that sounds like a potentially good idea even without it being able to do any calculations. You'll need to be very careful when you think about what to do if you're ever asked to convert a decimal value which you can't represent exactly though. I'd be interested in seeing such a type, to be honest. Hmm...

(As other answers have indicated, I'd really make sure that it's the numbers which are taking up the memory before doing this, however. If you don't need to do it, there's little point in introducing the extra complexity speculatively.)

Answer 6

Is "just add more memory" an acceptable answer?

How much cost is involved in properly coding and testing the proposed approach of moving the values between these representations. Compare that cost with shovelling more memory into a machine with the app running as a 64 bit process.

Answer 7

From MSDN decimal: There is no implicit conversion between floating-point types and the decimal type; therefore, a cast must be used to convert between these two types.

It looks like it is REQUIRED to do the cast in the case you are using.

That being said, it is very important that you understand what most other people here are posing about in regards to the problems of representing currency in floating point.

You may way to consider creating/finding a 64bit BCD (Binary Coded Decimal) implementation that you can use for your system.

Answer 8

As most of the other posts have already pointed out, converting between 128-bit decimal and 64-bit floating point representations is not a conversion that will always maintain accuracy.

However, if you are dealing with the prices of financial shares, you could consider representing them as ints (the number of pennies) rather than as decimal value (the number of fractional dollars). Perform all financial calculations in pennies and then only expose them to the outside world as decimals when requested.

Another approach may be to improve the algorithmic efficiency of your system rather than "compressing" the storage type. Do you really need all that data in memory at once? Can you virtualize it somehow?

If not, given the volume of data you are managing, you may want to look into organizing the data in a way the reduces the redundancy. For example, not every share has a historic price back in time (some companies don't exist far enough back in time). So orgnize your data as a dictionary of stock prices by day (or year) rather than as a tabular structure for each stock. There may be other alternatives depending on how your data is available and how you intend to perform calculations with it.