views:

36

answers:

1

I want to use a library that uses file descriptors as the basic means to access its data. For performance reasons, I don't want to have to commit files to the disk each before I use this library's functions.

I want to create (large) data blobs on the fly, and call into the library to send them to a server. As it stands, I have to write the file to disk, open it, pass the FD to the library, wait for it to finish, then delete the file on disk. Since I can re-create the blobs on demand (and they're not so large that they cause excessive virtual memory paging), saving them to disk buys me nothing, and incurs a large performance penalty.

Is it possible to assign a FD to a block of data that resides only as a memory-mapped entity?

+2  A: 

You could mount a memory-backed filesystem: http://lists.apple.com/archives/darwin-kernel/2004/Sep/msg00004.html

Using this mechanism will increase memory pressure on the system, and will probably be paged out if memory pressure is great enough. It might be worthwhile to make it a configuration option, in case the user would rather some other application have first-choice of the memory.

Another option is to use POSIX shared memory segments: http://opengroup.org/onlinepubs/007908799/xsh/shm_open.html (I haven't used POSIX shared memory segments myself; if I understand them correctly, they were designed to solve exactly this problem.)

The shm_open() function creates a memory object and returns a file descriptor. You could then mmap(2) that file descriptor, do your work, and pass the file descriptor to the library.

Don't forget to shm_unlink the object when you're done; POSIX shared memory segments, message queues, and semaphore arrays don't automatically go away when the last process exits.

sarnold
Aha! shm_open() is exactly what I need. Thanks!
Enquimot
@Mouviciel, thanks for the fix :D wish I could up-vote edits.
sarnold
May need to experiment with this, as while OS X supports POSIX, you may need to tweak kernel parameters to be able to assign reasonable amounts of shared memory. Depending on your application this may or may not be acceptable (i.e. I have no problem tweaking params to run postgres or Oracle, but most end users expect zero setup on install).
JulesLt
Thanks for the warning @JulesLt, some details here: http://www.network-theory.co.uk/docs/postgresql/vol3/SharedMemoryandSemaphores.html
sarnold