Python, ctypes and mmap

Question

I am wondering if it is possible for the ctypes package to interface with mmap.

Currently, my module allocates a buffer (with create_string_buffer) and then passes that using byref to my libraries mylib.read function. This, as the name suggests, reads data into the buffer. I then call file.write(buf.raw) to write the data to disk. My benchmarks, however, show this to be far from optimal (time spent in file.write is time better spent in mylib.read).

I am therefore interested in knowing if ctypes can interoperate with mmap. Given an mmap.mmap instance and an offset how can I get a pointer (c_void_p) into the address space?

Answer 1

Be aware that the operating system is going to be doing readahead for read() anyway. You're going to be blocking either in read() or write()--one or the other will bottleneck the operation--but even though you're blocking in one, that doesn't mean the other isn't taking place for you behind the scenes. That's the job of every multitasking operating system.

If you use mmap for this, you're very likely making things more complicated for the OS--making it harder for it to determine that you're really just streaming data in and out, and making it more complicated for it to do read-ahead. It may still figure it out (operating systems are very good at this), but you're probably not helping.

The only benefit in principle is avoiding the cost of a memory copy, but it doesn't sound like that's your goal here (and unless profiling clearly says otherwise, I'd strongly doubt that would help performance).

Answer 2

An mmap object "supports the writable buffer interface", therefore you can use the from_buffer class method, which all ctypes classes have, with the mmap instance as the argument, to create a ctypes object just like you want, i.e., sharing the memory (and therefore the underlying file) that the mmap instance has mapped. I imagine, in specific, that you'll want a suitable ctypes array.