precision

how to use "%f" to populate a double value into a string with the right precision

I am trying to populate a string with the double value using a sprintf like this sprintf(S, "%f", val); But the precision is being cut of to 6 decimal places. I need about 10 decimal places for the precision. Kindly tell me how that can be achieved. ...

How to manually parse a floating point number from a string

Of course most languages have library functions for this, but suppose I want to do it myself. Suppose that the float is given like in a C or Java program (except for the 'f' or 'd' suffix), for example "4.2e1", ".42e2" or simply "42". In general, we have the "integer part" before the decimal point, the "fractional part" after the decima...

Precision of reals through writeln/readln in Delphi

My clients application exports and imports quite a few variables of type real through a text file using writeln and readln. I've tried to increase the width of the fields written so the code looks like: writeln(file, exportRealvalue:30); //using excess width of field .... readln(file, importRealvalue); When I export and then import ...

2.9999999999999999 >> .5?

I heard that you could right-shift a number by .5 instead of using Math.floor(). I decided to check its limits to make sure that it was a suitable replacement, so I checked the following values and got the following results in Google Chrome: 2.5 >> .5 == 2; 2.9999 >> .5 == 2; 2.999999999999999 >> .5 == 2; // 15 9s 2.9999999999999999 >...

Why do I see a double variable initialized to some value like 21.4 as 21.399999618530273?

double r = 11.631; double theta = 21.4; In the debugger, these are shown as 11.631000000000000 and 21.399999618530273. How can I avoid this? ...

double in .net

If I have the following code (this was written in .NET) double i = 0.1 + 0.1 + 0.1; Why doesn't i equal 0.3? Any ideas? ...

Php precision with floating point - complex equation = 1

I have the following equation 1 - ((.5 * 0.83333333333333) ^ 2 + (.5 * 0.83333333333333) ^ 2 + (.5 * (1 - 0.83333333333333)) ^ 2 + (.5 * (1 - 0.83333333333333)) ^ 2) In Php5, this results in an answer of 1 as opposed to .63 (on two machines, OSx and Centos). Should I be exclusively using the bc math functions of Php to do equations l...

What is the standard (or best supported) big number (arbitrary precision) library for Lua?

I'm working with large numbers that I can't have rounded off. Using Lua's standard math library, there seem to be no convenient way to preserve precision past some internal limit. I also see there are several libraries that can be loaded to work with big numbers: http://oss.digirati.com.br/luabignum/ http://www.tc.umn.edu/~ringx004/m...

Retain precision with Doubles in java.

public class doublePrecision { public static void main(String[] args) { double total = 0; total += 5.6; total += 5.8; System.out.println(total); } } Which returns 11.399999999999 Okay clarifying the question a bit, how would i get this to just print (or be able to use it as) 11.4? ...

How to write a prototype for Number.toFixed in JavaScript?

I need to round decimal numbers to six places using JavaScript, but I need to consider legacy browsers so I can't rely on Number.toFixed The big catch with toExponential, toFixed, and toPrecision is that they are fairly modern constructs not supported in Mozilla until Firefox version 1.5 (although IE supported the methods since vers...

What units of measure would you store engineering data in?

In our app, we currently live with the legacy of a decision to store all engineering data in our database in SI. I worry that we may run the risk of not having sufficient precision and accuracy in our database or in .NET numeric types. I am also worried that we may see artifacts of floating-point maths (although that is probably a quest...

How best to sum up lots of floating point numbers?

Imagine you have a large array of floating point numbers, of all kinds of sizes. What is the most correct way to calculate the sum, with the least error? For example, when the array looks like this: [1.0, 1e-10, 1e-10, ... 1e-10.0] and you add up from left to right with a simple loop, like sum = 0 numbers.each do |val| sum += val...

Floating Point Limitations

My code: a = '2.3' I wanted to display a as a floating point value. Since a is a string, I tried: float(a) The result I got was : 2.2999999999999998 I want a solution for this problem. Please, kindly help me. I was following this tutorial. ...

How many digits can be stored for a number in SQL Server 2005?

I have a number from an Oracle database of 47306832975095894070.85314746810624532. When I bring it into SQL Server, it certainly doesn't show that many digits. It shows as 4.73068329750959E+19, and the field is defined as FLOAT. I think that probably includes all the significant digits, but I'm being asked if the number can be store...

Lost precision on GMP mpf_add. Where have my digits gone?

I'm summing two negative floats: char * lhs = "-2234.6016114467412141"; char * rhs = "-4939600281397002.2812"; According to Perl, using bignum and Math::BigFloat, the answer is -4939600281399236.8828114467412141 However, according to GMP, using the code below, the answer is -4939600281399236.88281 Where have I gone wrong? What h...

What is the precision of long double in C++?

Does anyone know how to find out the precision of long double on a specific platform? I appear to be losing precision after 17 decimal digits, which is the same as when I just use double. I would expect to get more, since double is represented with 8 bytes on my platform, while long double is 12 bytes. Before you ask, this is for Pro...

C#.NET: Is it safe to check floating point values for equality to 0?

I know you can't rely on equality between double or decimal type values normally, but I'm wondering if 0 is a special case. While I can understand imprecisions between 0.00000000000001 and 0.00000000000002, 0 itself seems pretty hard to mess up since it's just nothing. If you're imprecise on nothing, it's not nothing anymore. But I do...

Best algorithm for avoiding loss of precision?

A recent homework assignment I have received asks us to take expressions which could create a loss of precision when performed in the computer, and alter them so that this loss is avoided. Unfortunately, the directions for doing this haven't been made very clear. From watching various examples being performed, I know that there are cert...

javascript 64 bit numeric precision

Hi, Is there a way to represent a number with higher than 53 digit precision in javascript? In other words, is there a way to represent 64 bit precision number? I am trying to implement a logic in which each bit of a 64bit number represents something. I loose the lower significant bits when I try to set bits higher than 2^53. Math.pow(...

Using integers and requiring multiplication vs. using decimals as a data type - what are your thoughts?

What are your thoughts on this? I'm working on integrating some new data that's in a tab-delimited text format, and all of the decimal columns are kept as single integers; in order to determine the decimal amount you need to multiply the number by .01. It does this for things like percentages, weight and pricing information. For examp...