Suppose a hypothetical 6-bit floating point representation, with the fraction occupying 2 bits, and the exponent occupying 3 bits. What is the biggest (except for ) number this 6-bit floating point representation can support? I read this from the book "Computer System: A programmer's perspective" on page P71. I guess this to be 28, but ...
I'm trying to store a C# double in MS SQL 2005 as a float. However, there seem to be a range of small numbers which are valid doubles but which aren't valid floats. For example, when I try to store the value 1e-320 I get the error "the supplied value is not a valid instance of type float". This is consistent with the documentation for S...
I'm currently trying to get something to run involving trigonometry but I've run across a hitch involving the math.asin function (it also applies to acos and atan but in those cases it less affects what I'm trying to do). The issue is best summarised by two posts from a help thread I found about it elsewhere; Sorry, I have just tried...
Hi, I am writing code that will deal with currencies, charges, etc.. I am going to use the BigDecimal class for math & storage. We ran into something weird with it, however. Using this statement: 1876.8 == BigDecimal('1876.8') it returns false. If I run those values through a formatting string "%.13f" I get: "%.20f" % 1876.8 => ...
Duplicate: Why do I see a double variable initialized to some value like 21.4 as 21.399999618530273? Is JavaScript’s math broken? many, many other questions, for every language and platform, all with the same answer. trace( ">> " + (399.6-(Math.floor(399.6))) ); prints out >> 0.6000000000000227 why? ...
The following will ensure that any large numbers will only be precise to the hundredths place (related to this answer): public function round( sc:Number ):Number { sc = sc * 100; sc = Math.floor( sc ); sc = sc / 100; return sc; } What is the optimal way to round my numbers to the precision of .05? Is there something ...
I can name three advantages to using double (or float) instead of decimal: Uses less memory Faster because floating point math operations are natively supported by processors Can represent a larger range of numbers But these advantages seem to apply only to calculation intensive operations, such as those found in modeling software. O...
Is there a built-in library that I can use to convert c99 style floating point notation, example: 0x1.0p23f, to regular floating point numbers using Perl (and vice versa)? Thanks, ...
I'm trying to take two 7 digit numbers (source and target record counts) and calculate the percentage of missing records in the target count. For example, my record counts could be 4084094 for the source, and 4081313 for the target. I'm also using the follow formula to calc the percentage: ((1 - (target / source)) * 100) For my exampl...
I am coding several reference algorithms in both Java and C/C++. Some of these algorithms use . I would like for the two implementations of each algorithm to produce identical results, without rounding differently. One way to do this that has worked consistently so far is to use a custom-defined pi constant which is exactly the same in b...
I write line of business applications. I'd like to build a front-end end using Javascript and am trying to figure out how to deal with, for a business user, are floating point errors (I understand from a computer science perspective they might not be considered errors). I've read plenty on this and seen all kinds of rounding hacks that...
Right now I convert the string containing the decimal number to an integer (ignoring the radix point for now), load it into ST(0), and divide by the correct power of ten to account for the radix point. This seems round about, and requires I have a look up table for some of the powers of 10. Is there a better way to do this? ...
The following SAS code: data _null_; format t u best32.; t = 10000000000000000000000000; u = 1e25; put t u; if t ne u then put 'diff'; run; on my Windows machine prints out: 10000000000000000905969664 9999999999999998758486016 diff While I understand that only the first 15-16 digits are to be trusted, why do they give dif...
I am pretty formatting a floating point number but want it to appear as an integer if there is no relevant floating point number. I.e. 1.20 -> 1.2x 1.78 -> 1.78x 0.80 -> 0.8x 2.00 -> 2x I can achieve this with a bit of regex but wondering if there is a sprintf-only way of doing this? I am doing it rather lazily in ruby like so: ("...
Recently, sometimes (rarely) when we export data from our application, the export log contains float values that look like "-1.#J". I haven't been able to reproduce it so I don't know what the float looks like in binary, or how Visual Studio displays it. I tried looking at the source code for printf, but didn't find anything (not 100% s...
I recently discovered that x**.5 and math.sqrt(x) do not always produce the same result in Python: Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] on win32 >>> 8885558**.5 - math.sqrt(8885558) -4.5474735088646412e-13 Checking all integers below 10**7 produced an error rate of almost exactly 0.1%, with the s...
I've been doing some profiling lately and I've encountered one case which is driving me nuts. The following is a piece of unsafe C# code which basically copies a source sample buffer to a target buffer with a different sample rate. As it is now, it takes up ~0.17% of the total processing time per frame. What I don't get is that if I use ...
So, I know a little bit about how floating point are represented, but not enough to be sure of my answer. The general question: for a given precision (for my purposes, the number of accurate decimal places in base 10), what range of numbers can be represented for 16-, 32-, and 64-bit IEEE-754 numbers? Specifically, I'm only interested ...
in PHP echo (int) ( (0.1+0.7) * 10 ); or in Ruby p ((0.1+0.7) *10).to_i the result is 7 instead of 8. it might be really hard to catch for such pitfalls. i think if in North America, it is less of a problem because we calculate prices up to how many cents, so $17.28 or $17.29 might be less of a problem. But in country like Chin...
I have some C# unit tests that perform some float/double operations and I would like to unit test them. Assert.AreEqual is insufficient because of rounding errors. Take unit conversion as an example. 10.5 meters to feet has a conversion factor of 3.281 so I get 34.4505. Using a more accurate conversion factor gives me 34.4488189. I ...