HI i wrote a program by python , and when i open too many tempfile, i will got an exception: Too many open files ... Then i figure out that windows OS or C runtime has the file-handle limits, so, i alter my program using StringIO(), but still don`t know whether StringIO also is limited??
+6
A:
Python's StringIO does not use OS file handles, so it won't be limited in the same way. StringIO will be limited by available virtual memory, but you've probably got heaps of available memory.
Normally the OS allows a single process to open thousands of files before running into the limit, so if your program is running out of file handles you might be forgetting to close them. Unless you're intending to open thousands of files and really have just run out, of course.
Greg Hewgill
2009-07-24 12:08:33
oh,thank one's lucky stars The program is multi-threading as a message server,so some thread put messages into queue,others get them from the queue.As using file-handle (in case of memory lack, i choose file-handle first), but if there r too many message on the queue, means too many file-handles the program opened, then got exception.So, in case of that, i think the only way to handle this problem is using StringIO() instead of file-handles....Hope my boss is willing to buy more memory for it...:)
Andy
2009-07-24 12:19:20
What is in the files? Why do you need so many open at once? Why don't you just close the files? How is "the queue" implemented?
John Machin
2009-07-24 16:11:59
strings are in the files.reason is not what i need opening so many files, it`s for the queue problem...front thread puts objects with file-handles into queue, back threads get those objects from the queue and write the strings in tempfiles to DB (after that, close tempfiles of course..).As matter of the DB operation, front thread is >100 times faster than back threads, so, when program comes with rush hour, there will be many objects with file-handles stacked in the queue, then program must holds lots of tmpfiles at runtime...
Andy
2009-07-25 09:04:55