How should I share and store data in a small multithreaded python application?

Question

Hi all,

I'm writing a small multithreaded client-side python application that contains a small webserver (only serves page to the localhost) and a daemon. The webserver loads and puts data into a persistent "datastore", and the daemon processes this data, modifies it and adds some more. It should also takes care of the synchronization with the disk.

I'd like to avoid complicated external things like SQL or other databases as much as possible.

What are good and simple ways to design the datastore? Bonus points if your solution uses only standard python.

Answer 1

What you're seeking isn't too Python specific, because AFAIU you want to communicate between two different processes, which are only incidentally written in Python. If this indeed is your problem, you should look for a general solution, not a Python-specific one.

I think that a simple No-SQL key-value datastore such as Redis, for example, could be a very nice solution for your situation. Contrary to "complicated" using a tool designed specifically for such a purpose will actually make your code simpler.

If you insist on a Python-only solution, then consider using the Python bindings for SQLite which come pre-installed with Python. An SQLite DB can be concurrently used by two processes in a safe manner, as long as your semantics of data access are well defined (i.e. problems you have to solve anyway, the tool nonwithstanding).