How to share data between python processes without writing to disk.

Question

Helllo, I would like to share small amounts of data (< 1K) between python and processes. The data is physical pc/104 IO data which changes rapidly and often (24x7x365). There will be a single "server" writing the data and multiple clients reading portions of it. The system this will run on uses flash memory (CF card) rather than a hard drive, so I'm worried about wearing out the flash memory with a file based scheme. I'd also like to use less power (processor time) as we are 100% solar powered.

• Is this a valid worry? We could possibly change the CF card to a SSD.
• Does changing a value using mmap physically write the data to disk or is this a virtual file?
• We will be running on Debian so perhaps the POSIX IPC for python module is the best solution. Has anyone used it?
• Has anyone tried the Python Object Sharing (POSH) module? It looks promising at first glance but it is in "Alpha" and doesn't seem to be actively being developed.

Thank You

UPDATE: We slowed down the maximum data update rate to about 10 Hz, but more typically 1 Hz. Clients will only be notified when a value changes rather than at a constant update rate. We have gone to a multiple servers/multiple clients model where each server specializes in a certain type of instrument or function. Since it turned out that most of the programming was going to be done by Java programmers, we ended up using JSON-RPC over TCP. The servers wil be written in Java but I still hope to write the main client in Python and am investigation JSON-RPC implementations.

Answer 1

Create a ramfs partition and write to that. (You could use tmpfs, but unlike tmpfs, ramfs is not swapped to disk). However, as ramfs doesn't have a size limit, you must take care that you don't run out of memory; since you're only writing a tiny bit of data there, it shouldn't be a problem.

This way, your data won't ever be written to a disk (note: you will lose them if power fails).

See also the ramfs docs.

Answer 2

Anm alternative to writing the data to file in the server process might be to directly write to the client processes:

Use Unix domain sockets (or TCP/IP sockets if the clients run on different machines) to connect each client to the server, and have the server write into those sockets. Depending on your particular processing model, choosing a client/socket may be done by the server (e.g. round-robin) or by the clients signalling that they're ready for more.

Answer 3

According to the Wikipedia article about the mmap system call, memory mapped files contents are written back to disk when updated.

Have you looked at the multiprocessing module (in standard library) - especially the part Sharing state between processes?

Ramfs as mentioned by Piskvor also seems like a good solution - especially when not all processes are written in Python.

Answer 4

When running on flash systems, make sure your filesystem is designed properly to maximize the life of the flash memory (wear levelling). JFFS and, I believe, others are now capable of doing this effectively. If you use such a system, you shouldn't be overly concerned about using the flash, but certainly if you're writing a constant stream of data you'd want to avoid doing that on the flash.

Using a RAM filesystem is a good idea. Better yet is to avoid filesystems entirely if the system design will let you. To that end you mention POSH. I've never tried it, but we've found Pyro ("PYthon Remote Objects") to be an elegant and effective solution in some similar cases.

And of course there's the standard library multiprocessing module, which bears some similarities in terms of how it communicates between processes. I'd start there for any new development in this area, and go elsewhere only if it failed to pan out.