floating-accuracy

Large numbers erroneously rounded in Javascript

See this code: <html> <head> <script src="http://www.json.org/json2.js" type="text/javascript"></script> <script type="text/javascript"> var jsonString = '{"id":714341252076979033,"type":"FUZZY"}'; var jsonParsed = JSON.parse(jsonString); console.log(jsonString, jsonParsed); </script> </head> <body> </body> </html> Wh...

Building physical simulation (Bullet3d) for iPhone SDK 2.2 leads to different results than in SDK 3.0

We are building a simple rigid body 3D collision application using Bullet3D on the iPhone. When testing it, everything goes fine on the simulator and on the iPhone SDK 3.0. When trying it on the iPhone SDK 2.2 some rigid bodies pass through each other... any idea why? We assume it has to do with (the not verified) fact that Xcode uses g...

polynomial evaluation accuracy, multiplication versus division

let us say I have have polynomial in x, divided by a power of x: p = (a + x(b + x(c + ..)))/(x**n) efficiency aside, which would be more accurate computation numerically, the above or using division: p = (((a/x + b)/x + c)/x + ...) thanks ...

The Jquery displays $20.099999999999998 instead of $20.1

This is the Jquery code: function Totalprice() { var unitprice=$('#unitpay').val(); var quota=$('#readers').val(); var totalprice=unitprice*quota; $('#totalprice').text('$'+totalprice); } When the value of readers is 67 and the unitpay is 0.3, it calculates the total price and displays $20.099999999999998 , not $20....

Python float - str - float weirdness

>>> float(str(0.65000000000000002)) 0.65000000000000002 >>> float(str(0.47000000000000003)) 0.46999999999999997 ??? What is going on here and how do I convert 0.47000000000000003 to string and the resultant value back to float? I am using python 2.5.4 on windows. ...

How Decimal places are converted using FLOAT in SQL Server 2000/2005/2008

Hi, in this SO question the OP wanted to drop the 0's in the decimal places for his results. Now the example I gave (below) to get around this was to CAST with DECIMAL, then CAST with FLOAT. e.g. SELECT CAST(0.55 AS FLOAT) Using the example above and running it in SQL Server 2005/2008 would seem to bring up the correct result of 0.5...

How do I set the floating point precision in Perl?

Is there a way to set Perl script's floating point precision (to 3 digits), without having to change it specifically for every variable? Something similar to TCL's: global tcl_precision set tcl_precision 3 ...

Floating point formatting in printf()

i have array of floats where data are stored with varying decimal points so some are 123.40000, 123.45000, 123.45600...now if i want to print these values in the string without the 0s in the end in printf() so that they are 123.4, 123.45, 123.456, without those 0s in the end. is this possible? if so how? ...

Can 12.1 be represented exactly as a floating point number?

This is in reference to the comments in this question: This code in Java produces 12.100000000000001 and this is using 64-bit doubles which can present 12.1 exactly. – Pyrolistical Is this true? I felt that since a floating point number is represented as a sum of powers of two, you cannot represent 12.1 exactly, no matter how many...

Integers and float precision

This is more of a numerical analysis rather than programming question, but I suppose some of you will be able to answer it. In the sum two floats, is there any precision lost? Why? In the sum of a float and a integer, is there any precision lost? Why? Thanks. ...

sprintf(buf, "%.20g", x) // how large should buf be?

I am converting double values to string like this: std::string conv(double x) { char buf[30]; sprintf(buf, "%.20g", x); return buf; } I have hardcoded the buffer size to 30, but am not sure if this is large enough for all cases. How can I find out the maximum buffer size I need? Does the precision get higher (and theref...

Validate double value range and step

I'm trying to construct an algorithm that validates that a double value is a member of a range defined with min, max and step values. The problem is checking that the value obeys the step rule. For integers this can be easily done: boolean validate(int value, int min, int step, int max){ return value >= min && value ...

Why there is a discrepancy in the result?

If I apply Binet Formula and Recursive formula for finding the fibonaci series, there is a discrepancy in result. Why? Basically I am a student and it is our assignment to implement the fibbonacci series. So I while making the experiment I came across this situation. Thanks in advance ...

Is (1 + sqrt(2))^2 = 3 + 2*sqrt(2) satisfied in Floating Point arithmetics?

In mathematics the identity (1 + sqrt(2))^2 = 3 + 2*sqrt(2) holds true. But in floating point (IEEE 754, using single precision i.e. 32 bits) calculations it's not the case, as sqrt(2) doesn't have an exact representation in binary. So does using a approximated value of sqrt(2) provide different results for left and right hand sides? I...

Are there compilers that optimise floating point operations for accuracy (as opposed to speed)?

We know that compilers are getting better and better at optimising our code and make it run faster, but my question are there compilers that can optimise floating point operations to ensure greater accuracy. For example a basic rule is to perform multiplications before addition, this is because multiplication and division using floating...

Why don't I get zero when I subtract the same floating point number from itself in Perl?

Possible Duplicates: Why is floating point arithmetic in C# imprecise? Why does ghci say that 1.1 + 1.1 + 1.1 > 3.3 is True? #!/usr/bin/perl $l1 = "0+0.590580+0.583742+0.579787+0.564928+0.504538+0.459805+0.433273+0.384211+0.3035810"; $l2 = "0+0.590580+0.583742+0.579788+0.564928+0.504538+0.459805+0.433272+0.384211+0.3035810"; $...

Floating point inaccuracy examples

How do you explain floating point inaccuracy to fresh programmers and laymen who still think computers are infinitely wise and accurate? Do you have a favourite example or anecdote which seems to get the idea across much better than an precise, but dry, explanation? How is this taught in Computer Science classes? ...

Counting digits in a float

I am following some beginner tutorials for OpenGL in c++ but as I started off as a c# programmer it has made me take a lot of things for granted. So my problem occurred when I was debug printing my FPS reading to the output. I think the method was something like DebugPrintString off the top of my head which took a char* and basically i w...

How to convert strings to floats with perfect accuracy?

I'm trying to write a function in the D programming language to replace the calls to C's strtold. (Rationale: To use strtold from D, you have to convert D strings to C strings, which is inefficient. Also, strtold can't be executed at compile time.) I've come up with an implementation that mostly works, but I seem to lose some precisi...

Big float for shader-based mandelbrot explorer

Hi All, I've managed to create a simple mandelbrot explorer using Open Gl, and the CGFX SDK provided by NVidia. It works well, but is currently float based, and therefore doesn't have much "depth" -- As the distance from the lowest complex number to the largest becomes smaller, the precision is lost, and the resultant image is "pixelate...