I have a Spring application running against a MySql database. I have a sales table like: +-------------------+------------------------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------------+------------------------------+------+-----+---------+-----...
I encountered a weird behaviour in Perl. The following subtraction should yield zero as result (which it does in Python): print 7.6178E-01 - 0.76178 -1.11022302462516e-16 Why does it occur and how to avoid it? P.S. Effect appears on "v5.10.0 built for x86_64-linux-gnu-thread-multi" (Ubuntu 9.04) and "v5.8.9 built for darwin-2level" (...
SQL Sever 2000 documentation: Is a floating point number data with the following valid values: –3.40E + 38 through -1.18E - 38, 0 and 1.18E - 38 through 3.40E + 38. Storage size is 4 bytes. In SQL Server, the synonym for real is float(24). SQL Server 2005 documentation: The ISO synonym for real is float(24). EDIT: G...
I'm trying to use the bignum module in Perl and want to set the precision. I know this can be done via a one liner as detailed on the module's CPAN page: $ perl -Mbignum=p,-50 -le 'print sqrt(20)' ...which will print out the square root of 20 to 50 digits of precision, but what I'm wondering is if there's anyway to set the precision ...
I just ran into some unexpected behavior with DateTime.UtcNow while doing some unit tests. It appears that when you call DateTime.Now/UtcNow in rapid succession, it seems to give you back the same value for a longer-than-expected interval of time, rather than capturing more precise millisecond increments. I know there is a Stopwatch cl...
Hello, I'm working with C# and SQL 2005. I've stored a float variable of the value 0.2 into table column of real type. When I opened the table to check the values, I found a value of 0.200000029... It is known problem, real is approximate type. However, I've changed the value using management studio to 0.2 and reloaded the table. The dat...
Hi I'm writing a program in C# which suppose to calculate very small numbers. However, all my calculations are getting NaN value because they are too small for the 'double' datatype. Is there an option in C# to deal with such situations? Thanks, Eden. ...
Hi folks, I am developing a mathematical parser which is able to evaluate String like '5+b*sqrt(c^2)'. I am using ANTLR for the parsing and make good progress. Now I fell over the Java class BigDecimal and thought: hey, why not thinking about precision here. My problem is that the Java API does not provide trigonometric methods for Big...
Hi, I wonder if there is something like eps to represent the value of machine precision in C++? Can I use it as the smallest positive number that a double can represent? Is it possible to use 1.0/eps as the max positive number that a double can represent? Where can I find eps in both C++ and C standard libraries? Thanks and regards! ...
Is it even that inaccurate? I re-implented the whole thing with Apfloat arbitrary precision and it made no difference which I should have known to start with!! public static double bearing(LatLng latLng1, LatLng latLng2) { double deltaLong = toRadians(latLng2.longitude - latLng1.longitude); double lat1 = toRadians(latLng1.latitude); ...
I've encountered an annoying problem in outputting a floating point number. When I format 11.545 with a precision of 2 decimal points on Windows it outputs "11.55", as I would expect. However, when I do the same on Linux the output is "11.54"! I originally encountered the problem in Python, but further investigation showed that the diff...
I'm making a game with Python->PyGame->Albow and ran into a problem with board generation. However I'll try to explain the problem in a language agnostic way. I believe it's not related to python. I've split the game board generation into several parts. Part one generates the board holes. Holes are contained in a list/array. Each hole...
I wish to round a floating point number to set precision and return the result from a function. For example, I currently have the following function: inline bool R3Point:: operator==(const R3Point& point) const { // Return whether point is equal return ((v[0] == point.v[0]) && (v[1] == point.v[1]) && (v[2] == point.v[2])); } What ...
Is there any difference in precision between one time assignment: res=n/k and multiple assignment in for cycle: for i in range(n): res+=1/k ? ...
I have the following column specified in a database: decimal(5,2) How does one interpret this? According to the properties on the column as viewed in SQL Server Management studio I can see that it means: decimal(Numeric precision, Numeric scale). What do precision and scale mean in real terms? It would be easy to interpret this as a ...
When I call get_d() on a MPQ variable in the GMP library, I only get at most six digits. C++ is supposed to provide 8 bytes or ~15 digits for doubles. Why am I only getting six digits? ...
Hello all, I have a problem concerned with losing of precision my task is to print numbers as strings int exponent = ... int[] Mantissas = { 1, 2, 5 }; double dataStep = java.lang.Math.pow(10.0, exponent) * Mantissas[mantissaIndex]; ... for (int i = 0; i < NSteps; i++) steps[i] = firstStep + i * dataStep; draw(ste...
Hi, I'm just pondering about whether or not it's possible to create one's own data type? So if you need more precision, that's not supported by one of the basic types, you could just "create" your own fullfilling your requierements. Is that possible and how? ...
This seems to have been asked before: http://stackoverflow.com/questions/1402547/rails-decimal-precision-and-scale But when running a change_column migration for :precision or :scale they don't actually affect the schema or database, but db:migrate runs without errors. My migration file looks like this: class ChangePrecisionAndScaleO...
I have a TDateTime variable which is assigned a value at runtime of 40510.416667. When I extract the time to a TTime type variable using the Frac function, it sets it to 0.41666666666. Why has it changed the precision of the value and is there a workround to retain the precision from the original value ie. to set it to 0.416667. ...