Scratch disks in Python?

Question

I understood that in certain Windows XP programs, like Photoshop, there is something called "scratch disks". What I understood that this means, and please correct me if I'm wrong, is that Photoshop manages its own virtual memory on the hard-drive, instead of letting Windows manage it. I understood that the reason for this is some limitation by Windows XP on how much total memory a process can take, regardless of HD space. I think it's around 3 GB. Did I get it right so far?

I am making an application in Python for running simulations. It will take a lot of memory, and will run on Windows XP. Is it possible for it to use scratch disks? How?

Answer 1

Scratch disks will benefit your application in the case that it works with very big files,

Is that the case?

If not, then i don't think you may find something that will benefit your application in scratch disks.

Answer 2

Until you ACTUALLY run out of memory, thinking about this is a waste of time.

When you finally do run out of memory, you'll need to use a temporary file to store objects that your process needs, but can't fit into memory.

Use pickle or shelve (see Data Persistence) your objects in a file. If that file happens to be on a disk named "scratch", well that's nice.

Sometimes you want your temporary files to be on a separate disk from your other working files for performance reasons. In some environments (SAN, NAS, storage arrays) your disks are virtual and looking for a "scratch" disk doesn't have any performance benefit. In other environments (i.e., you own all the hardware) you can put temporary files on some other drive, making that drive a "scratch" disk.

Answer 3

Memory mapped files might be what you are looking for. Python's implementation lets you use a file like a mutable string in memory.

Answer 4

I understood that the reason for this is some limitation by Windows XP on how much total memory a process can take, regardless of HD space. I think it's around 3 GB.

Just an FYI, this is more a limitation of a 32-bit OS rather than being a Windows XP problem. You'll have the same problem in 32-bit Vista, linux, bsd... you get the idea. If you go the 64-bit route, you don't have these problems.

For example, Windows XP x64 allows up to 8 terabytes of memory per process.

Answer 5

The Win32 API provides this: link text. You may be able to use these functions through PyWin32.

Answer 6

You could combine S.Lott's answer about using pickle (you should use cPickle though for better performance) with SqlLite.

sqlite is built into python 2.5 and up, so all you'll need to do is import :), then just store the pickled objects as strings in there and you'll have a nice fast method of accessing the data (compared to building your own method) that will help keep you organized as well.

note: cPickle is almost identical to pickle in use. Only difference is that it is written in C

Useful Python Docs:

• sqlite3 module
• pickle module

edit: It may be a good idea to have a user controlled memory usage limit. It would be a shame to be storing a bunch a data on disk and waiting on slow-ass disk I/O when the user has 8GB of RAM ;)