floating-point

Why does toPrecision return a String?

View this code: function testprecision(){ var isNotNumber = parseFloat('1.3').toPrecision(6); alert(typeof isNotNumber); //=> string } I would have expected a number. If 'isNotNumber' should be a real number, recasting is the solution: alert(typeof parseFloat(isNotNumber)) //=> number [Edit] thanks for your answers. Precisi...

PI and accuracy of a floating point number

A single/double/extended precision floating point representation of Pi is accurate up to how many decimal places? ...

Why doesn't C have unsigned floats?

I know, the question seems to be strange. Programmers sometimes think too much. Please read on... In C I use signed and unsigned integers a lot. I like the fact that the compiler warns me if I do things like assigning a signed integer to an unsigned variable. I get warnings if I compare signed with unsigned integers and much much more. ...

When to use "strictfp" keyword in java?

Ok, I've looked up what this does, but does anyone actually have an example of when you would use the "strictfp" keyword in java? Has anyone actually found a use for this? Would there be any side-effects of just putting it on all my floating point operations? ...

Floating point number in JS

I'm writing a small webpage that will enable students to answer questions and get feedback on their answers. Part of this detection checks for common errors to give them guidance. Specifically I want to check if their answer is a power of ten out from the actual answer. If the answer was 3.93E-6, this condition should activate if they ...

Floating Point errors in Colt Java matrix libraries

How do I avoid floating point errors in financial calculations performed with Colt matrix libraries? ...

How do I print a double value with full precision using cout?

So I've gotten the answer to my last question (I don't know why I didn't think of that). I was printing a double using cout that got rounded when I wasn't expecting it. How can I make cout print a double using full precision? Thanks ...

Visual C++ Data Breakpoints on Floating Points Codes Changing

Hello. Are there possibility to set breakpoint (as i see Data Breakpoint) in VS 2005(C++) on Floating Point codes(ST0-ST7) changing? For example on changing of value ST7 with condition ST7==INF. Thanks a lot. ...

Plese help me write a function to determine if two numbers are nearly equal when rounded to n significant decimal digits

I have been asked to test a library provided by a 3rd party. The library is known to be accurate to n significant figures. Any less significant errors can safely be ignored. I want to write a function to help me compare the results: def nearlyequal( a, b, sigfig=5 ): The purpose of this function is to determine if two floating-point n...

Can I force java to throw an error when dividing by zero with floating point numbers?

I wrote a simulator that has some collision detection code and does a good bit of math on each object when it detects collisions. If these two objects are at the exact same location or in some rare other cases, I'm getting NaN (not a number) as their location somewhere along the line and I'd like to know where. Normally, the program w...

How do display functions like sprintf convert numbers to strings?

I'm trying to extract the integer and decimal parts of a floating point value, and I seem to be running into some strange rounding problems, due probably to the imprecise way floats are stored. I have some code like this to extract the fractional part: double number = 2.01; int frac = int(floor(number * 100)) % 100; However the resu...

Is there a proximity map algorithm or data structure?

A problem I keep running into when writing code that draws images of scientific data is the following: Given some floating point data, fit those data into slots (1-dimensional case) or a grid (2-dimensional case) such that each datum is in the slot or grid entry whose value is closest to the datum's value. It is not the case that the s...

Is there a way to get a hashcode of a float with epsilon?

It is well known that comparing floats by == is usually a mistake. In a 3D-vector class (with float components X, Y, Z) i wrote, two vectors are considered equal if their distance is considered zero. public override bool Equals(object obj) { if (obj == null) { return false; } if (GetType () != obj.GetType ()) { return fal...

Is JavaScript's Math broken?

0.1 + 0.2 == 0.3 // returns false 0.1 + 0.2 // returns 0.30000000000000004 Any ideas why this happens? ...

Dealing with accuracy problems in floating-point numbers

I was wondering if there is a way of overcoming an accuracy problem that seems to be the result of my machine's internal representation of floating-point numbers: For the sake of clarity the problem is summarized as: // str is "4.600"; atof( str ) is 4.5999999999999996 double mw = atof( str ) // The variables used in the columns...

What operation turns floating point numbers into a "group" ?

Might anyone be famiiar with tricks and techniques to coerce the set of valid floating point numbers to be a group under a multiplication based operation? That is, given any two floating point numbers ("double a,b"), what sequence of operations, including multiply, will turn this into another valid floating point number? (A valid floa...

Java floating point math - (conversion for feet/meters)

Pretty basic question I think - I'm performing this function: private double convertMetersToFeet(double meters) { //function converts Feet to Meters. double toFeet = meters; toFeet = meters*3.2808; // official conversion rate of Meters to Feet return toFeet; } Problem is the output; for example I get 337.360800000...

What are the applications/benefits of an 80-bit extended precision data type?

Yeah, I meant to say 80-bit. That's not a typo... My experience with floating point variables has always involved 4-byte multiples, like singles (32 bit), doubles (64 bit), and long doubles (which I've seen refered to as either 96-bit or 128-bit). That's why I was a bit confused when I came across an 80-bit extended precision data type ...

What is the most reliable way of checking if a floating point variable is an integer?

I can think of several ways, eg. Convert.ToInt32(floatingPoint) - floatingPoint == 0; Math.Truncate(floatingPoint) - floatingPoint == 0; floatingPoint % 1 == 0; Math.Floor(floatingPoint) == floatingPoint; //etc... But which method is most reliable? ...

Floating point rounding when truncating

This is probably a question for an x86 FPU expert: I am trying to write a function which generates a random floating point value in the range [min,max]. The problem is that my generator algorithm (the floating-point Mersenne Twister, if you're curious) only returns values in the range [1,2) - ie, I want an inclusive upper bound, but my...