So I've decided to try to solve my physics homework by writing some python scripts to solve problems for me. One problem that I'm running into is that significant figures don't always seem to come out properly. For example this handles significant figures properly: from decimal import Decimal >>> Decimal('1.0') + Decimal('2.0') Decima...
Hi, I am converting an excel sheet formula to java but I can't understand how excel manages to take the following: 0.22 Applies a formula: =TEXT(R5/14, "h:mm") and somehow arrives at: 0.22 Again if I provide: 2.8 it arrives at 4.48 Can someone please explain to me how it does this. I have read a little regarding decimal and I understand...
What I'd like is a method to convert a double to a string which rounds using the half-up method. I.e. if the decimal to be rounded is a 5, it always rounds up the previous number. This is the standard method of rounding most people expect in most situations. I also would like only significant digits to be displayed. That is there should...
I have a problem similar to the one found here : http://stackoverflow.com/questions/86531/jsf-selectitem-label-formatting. What I want to do is to accept a double as a value for my and display it with two decimals. Can this be done in an easy way? I've tried using but that seems to be applied on the value from the inputText that is...
The number is bigger than int & long but can be accomodated in Decimal. But the normal ToString or Convert methods don't work on Decimal. ...
Hi, I have this situation. I have a real stored in a varbinary field in a sql 2005 database. As I can't convert a varbinary to a real in sql 2005, I'm trying to do that in vb.net. That field gets stored as a byte() array in a DataTable. Now I would like to read that byte() into a double, or decimal variable. But I don't have much of a ...
I wanted to see if folks were using decimal for financial applications instead of double. I have seen lots of folks using double all over the place with unintended consequences . . Do you see others making this mistake . . . ...
0x34363932353433373538323038353135353439 ...
I want to do this using the Math.Round function ...
Currently i have this method: static boolean checkDecimalPlaces(double d, int decimalPlaces){ if (d==0) return true; double multiplier = Math.pow(10, decimalPlaces); double check = d * multiplier; check = Math.round(check); check = check/multiplier; return (d==check); } But this method fails for chec...
I seem to be losing a lot of precision with floats. For example I need to solve a matrix: 4.0x -2.0y 1.0z =11.0 1.0x +5.0y -3.0z =-6.0 2.0x +2.0y +5.0z =7.0 this is the code i use to import the matrix from a text file: f = open('gauss.dat') lines = f.readlines() f.close() j=0 for line in lines: bits = string.split(line, ',') ...
I'm supporting/enhancing a web application written in Classic ASP/VBScript. It has been about 10 years since I have used either in a day to day capacity. I just ran across an issue that I would consider a "gotcha" and was wondering if others had similar things that I should learn to be aware of. My issue: I had to convert a Column in ...
Why does the select statement return two different values? declare @tempDec decimal set @tempDec = 1.0 / (1.0 + 1.0) select @tempDec, 1.0 / (1.0 + 1.0) ...
Python Decimal doesn't support being constructed from float, it expects that you have to convert float to a string first. This is very inconvenient since standard string formattors for float require that you specify number of decimal places rather than significant places. So if you have a number that could have as many as 15 decimal ...
I always tell in c# a variable of type double is not suitable for money. All weird things could happen. But I can't seem to create an example to demonstrate some of these issues. Can anyone provide such an example? (edit; this post was originally tagged C#; some replies refer to specific details of decimal, which therefore means System....
I write financial applications where I constantly battle the decision to use a double vs using a decimal. All of my math works on numbers with no more than 5 decimal places and are not larger than ~100,000. I have a feeling that all of these can be represented as doubles anyways without rounding error, but have never been sure. I would...
Hi folks, I'm doing some really simple math and saving the result to a MS SQL2008 DB. I'm averaging out the some numbers, which are byte values between 1<->5. I wish to record probably 2 decimal places only. I don't care about rounding for the 2nd decimal place (eg. a 1.155 == 1.5 or 1.6 .. i'm not too phased). So .. should i store th...
I am considering writing two limited precision alternatives to BigDecimal, namely DecimalInt and DecimalLong. These would be capable of dealing with numbers within the real bounds of int and long with an arbitrary number of decimal places, creatable in both mutable and immutable form. My plan is to make DecimalInt support +/-999,999,99...
Hi all, I do my php work on my dev box at home, where I've got a rudimentary LAMP setup. When I look at my website on my home box, any numbers I echo are automatically truncated to the least required precision. Eg 2 is echoed as 2, 2.2000 is echoed as 2.2. On the production box, all the numbers are echoed with at least one unnecessar...
I am sure this is very simple problem, but I am new to vb.net so am having an issue with it. I have a Decimal variable and I need to split it into two seperate variables, one containing the integer part, and one containing the fractional part. eg for x = 12.34 you would end up with a y = 12 and a z = 0.34 Are there any nice built in ...