views:

65

answers:

2

consider the function below:

int Func(char* filename);
int Func(FILE* filepointer);

these two do the same, reads alot of data from the given file (by name or pointer), analyze he result, and returns it. I wanna call this function with lots of different data. Therefore I should write data into file, then pass the new filename to Func. but data is huge and reading and writing in hard is very slow. actually the analyze time is much less than I/O.

can I get rid of save/load data all the time by any means? for example by making a FILE* pointer which points somewhere in Memory?

Update: obviously I don't have the source code of Func! It's a DLL call.

+2  A: 

You could use memory-mapped file technique or something like boost::iostreams with custom memory sinks / sources.

Actually, the second variant is a lot more flexible, but sometimes all that flexi- and versatibility is simply not needed.

Kotti
A: 

In many operating systems you can use an in-memory filesystem such as tmpfs -- and in Windows "temporary files" (opened with the appropriate flags, then rewound rather than closed) behave similarly (i.e., can stay in memory).

However, there isn't all that much to be gained there compared to writing (with lots of buffering) and reading (ditto) sequentially from an un-fragmented disk, for large files -- tmpfs's performance advantages are mostly for small files. If your performance is very bad, either the disk is horribly fragmented, or (perhaps more likely these days of self-adjusting filesystems) you're not using buffering appropriately (possibly just not buffering enough). (of course, both factors could be in play). Modern devices and filesystems can have awesome performance when just streaming huge buffers to and from memory, after all.

For a given amount of RAM devoted to buffering, you can get better performance (for what from app level look like huge numbers of tiny writes and reads) if that RAM is in userland in your app's address space (rather than under kernel control e.g. in a tmpfs), simply because you'll need fewer context switches -- and switches from user to kernel mode and back tend to dominate runtime when the only other ops performed are copies of small amounts of memory back and forth. When you use very large buffers in your app's stdio library, your "I/O" amounts to userland memory-memory copies within your address space with very rare "streaming" ops that actually transfers those buffers back and forth.

Alex Martelli