I'm currently using the charting within .NET using System.Windows.Forms.DataVisualization.Charting.Chart. Thus far it seems very powerful, and works great. However, there is a huge problem in terms of how it is auto-calculating intervals. I use a lot of double values, and in libraries like ZedGraph, it handles this perfectly. It selects ...
While learning about precision in floating point arithmetic and different methods to avoid it (using a conjugate, taylor series,...) books frequently mention the subtraction of two very similar numbers or one large and one small number as the biggest cause of error. How come it is only subtraction that causes this and not addition? As I ...
Do you know how to fix the following issue with math precision? p RUBY_VERSION # => "1.9.1" p 0.1%1 # => 0.1 p 1.1%1 # => 0.1 p 90.0%1 # => 0.0 p 90.1%1 # => 0.0999999999999943 p 900.1%1 # => 0.100000000000023 p RUBY_VERSION # => "1.9.2" p 0.1%1 # => 0.1 p 1.1%1 # => 0.10000000000000009 p 90.0%1 # => 0.0 p 90.1%1 # => 0.099999999999994...
Below is an excerpt from a program - float val=214.20; double val1=214.20; printf("\n\n\nfloat :: %f, %4.6f, %4.2f \n ",val,val,val); printf("double:: %f, %4.6f, %4.2f \n ",val1,val1,val1); And the output is - float :: 214.199997, 214.199997, 214.20<--- is the correct value as we wanted double:: 214.200000, 214.200000, 214.20...
Hi all, Having a bit of a problem here on a matrix multiplication code. I seem to lose precision on large matrices multiplications (my code runs fine on small matrices). My loop is the following : for (int j = 0; j < columns; j++) { float[] column = otherMatrix.Column(j); for (int i = 0; i < rows; i++) { double s =...
Trying out a problem of finding the first k digits of a num^num I wrote the same program in C++ and Python C++ long double intpart,num,f_digit,k; cin>>num>>k; f_digit= pow(10.0,modf(num*log10(num),&intpart)+k-1); cout<<f_digit; Python (a,b) = modf(num*log10(num)) f_digits = pow(10,b+k-1) print f_digits Input 19423474 9 Output...
I know that the absolute precision of position detection for gps hardware isn't that accurate. But assume that I have two androids and they are syncing each other at one place. In which precision (horizontal area) can I predict the other android position if the devices don't walk away too much (e.g. 1km) ? Would this as bad as several...
I want to store currencies in my (sqlite and mysql) databases. I'm trying to use the decimal column type, with :scale => 2. This isn't behaving as expected. If I save a record with :rate => 10.50, it is stored in my sqlite db as 10.5. In addition, when I output the value in a form field, it is displayed as 10.5. I don't want to do hack...
When rendering huge visuals in WPF, the visual gets distorted and more distorted with increasing coordinates. I assume that it has something to do with the floating point data types used in the render pipeline, but I'm not completely sure. Either way, I'm searching for a practical solution to solve the problem. To demonstrate what I'm t...
I am doing some floating point arithmetic and having precision problems. The resulting value is different on two machines for the same input. I read the post @ http://stackoverflow.com/questions/3031143/why-cant-i-multiply-a-float and also read other material on the web & understood that it is got to do with binary representation of floa...
In ActionScript 3 (and IIRC 2), the x and y properties on a display object are always stored as multiples of 0.05. so something like obj.x = 66.6666 is the same as obj.x = 66.65 Most of the time, this doesn't matter. But sometimes I can end up with really slow-moving objects, eg 1 pixel per second. 1/60 (fps) = 0.017 pixels per frame. ...