This error should be a simple one but I cant seem to make it work. The problem lies in the fact that this very same code works earlier in the program. I don's see any reason for it to be sending an error on this instance and not the four previous ones. Reference the code below, and feel free to provide any criticism you may have as it sh...
The .net framework provides in the Math class a method for powering double. But by precision requirement I need to raise a decimal to a decimal power [ Pow(decimal a, decimal b) ]. Does the framework have such a function? Does anyone know of a library with this kind of function? ...
If I have numbers as a text string without decimal places, how can I convert it to 2 decimal places such that 12345 is converted to 123.45? The string can be any length greater than 1. ...
I'm trying to optimize the following. The code bellow does this : If a = 0.775 and I need precision 2 dp then a => 0.78 Basically, if the last digit is 5, it rounds upwards the next digit, otherwise it doesn't. My problem was that 0.45 doesnt round to 0.5 with 1 decimalpoint, as the value is saved as 0.44999999343.... and setprecisi...
I have a database that contains decimals. The column is set up as ".. decimal(5,2) not null. What I'm trying to do is to find the sum of the column and displaying it in a text field by using this code.. public int getTotal() { Cursor cursor = sqliteDatabase2.rawQuery( "SELECT SUM(thedata) FROM thetable", null); if(cursor...
The problem is this, when I add two or more doubles from a table to a view, instead of giving me the right results, it adds a about ten or so more digits. For example 0.5+1.5=1.99999999998 or 5.5+8.5=14.0000000001. Any ideas? (I know this is sort of n00b question and I remember having to deal with stuff like that in the exams at 9th grad...
I am writing an application which does a lot of manipulation with decimal numbers (e.g. 57.65). As multiplications and divisions quickly erode their accuracy, I would like to store the numbers in a class which preserves their accuracy after manipulation, rather than rely on float and double. I am talking about something like this: clas...
How can I round a float (such as 37.777779) to two decimal places (37.78) in C? ...
I want to round off my decimal values as below: 7,715.625 becomes 7,716 How do I achieve this? ...
I'm working with MS Access to do some math. I'm taking a Cost and Dividing it by a decimal value to get a Price. I'm using a link table and a access sql query. SQL Ex Select (cost/markup) As Price From T_Cost; Calulcation Ex. 1234 / .55 = 2243.6363 1000 / .50 = 2000 I'm trying to figure out a way to remove the decimal places that w...
Not sure if that is the right way to ask this or not but here is the problem. Given a latitude of 26.746346081599476, how do I find the number 26.75 as the 16th greater than the number and 26.6875 as the 16th lower than the number? 26.0 26.0625 26.125 26.1875 26.25 26.3125 26.375 26.4375 26.5 26.5625 26.625 26.6875 My Number: 26.746346...
I have a model with field defined as p = models.DecimalField(max_digits=12, decimal_places=6) But the decimal_places=6 is only for internal accuracy. I only want 2 decimal_places to show up in the admin fields. So I thought perhaps I could do something with properties. But properties don't show up as fields in the Admin interface. E.g...
We're currently using System.Decimals to represent numbers in a .NET application we're developing. I know that decimals are design to minimize errors due to rounding, but I also know that certain numbers, 1/3 for example, cannot be represented as a decimal so some calculations will have small rounding error. I believe the magnitude of th...
I am very confused about how decimal numbers work in perl. I'm having trouble multiplying an int with a double. Here's what I have: sub timeEstimate(){ $number = shift; print "Number: $number\n"; $stuff = sprintf("%d", $number * $number * $number) * .2045; print "stuff: $stuff\n"; $totalDownloads = $stuff + ($number * $number) + $number...
I have a variable i would like to force to have 2 and always 2 decimals. Im comparing to a currency. Often i get a comparison looking like the following. if self.price != price #do something end Where self.price = 120.00 and price = 120.0. The self.price is set with a :precision => 2 in the model, but how do i do the same with a var...
Possible Duplicate: Is JavaScripts Math broken? Why does JS screw up this simple math? document.write(.1 + .2) // 0.3000000000000004 document.write(.3 + .6) // 0.8999999999999999 The first example is greater than the correct result, while the second is less. ???!! How do you fix this? Do you have to always convert decimal...
Hello, I have amethod which returns numbers like this: public decimal GetNumber() { return 250.00m; } Now when this value is printed to the console for example, it has a comma (250,00) instead of a point (250.00). I always want a point here, what am I doing wrong? Thanks ...
I'm trying to use Decimal.quantize() to achieve the following: - For any amount of money, expressed as a python decimal of default precision, I want to round it using decimal.ROUND_HALF_UP so that it has no cents after rounding. For example, given 1.25, I'm trying to obtain 1.00 (signifying no cents) given 1.49 I'm trying to obtain 1....
I have this block of code executed when pressing a number: NSString *currentValue = [@"" stringByAppendingFormat:@"%.02f", [[[[[textField text] stringByReplacingOccurrencesOfString:@"." withString:@""] stringByReplacingOccurrencesOfString:@"," withString:@""] stringByReplacingOccurrencesOfString:@" " withString:@""] doubleValue]/100...