What tools would you used to write a modular database in Python?

Question

I am building a modular application in python. The plugins are not known until runtime.

I envisage that many plugins will create/read/update data, which they themselves care about, and other plugins may care about too.

An approach I am considering:

• one plugin may be concerned with Users.
• Another plugin may be interested in one or more email addresses for those users.
• The Users plugin would expose the data, but the email data would not be available for joins outside .

What strategies/tools would you suggest I start constructing and using a modular database? Is this something an ORM tool could cater for? Is there any other magic I haven't considered?

I am envisaging that a side effect of a plugin architecture will make it more likely that the data will end up fairly normalized.

Edit:

• by plugins, I mean that the consumers of the plugins do not know about them upfront - they don't import them, rather the host plugins discover the extension plugins, or have them injected in.
• I am not looking for a database engine/ORM tool that is modular, I am looking to allow plugins to create a modular data model, which can be added to as more plugins are found.
• I am happy to write my own, but don't want to badly re-implement something that is already existing and much better
• I am happy to write my own, but don't want to implement a weak design where much stronger ones are more obviously suited to Python.

Answer 1

Python is already a dynamic programming environment. The imports are not resolved until run time. Python libraries are already a series of plugins. The essential "duck-typing" means that any compatible version of a library can be plugged in at run time.

You don't need to add any more dynamic stuff than Python already has. It's already completely dynamic, completely configured at run time.

"What strategies/tools would you suggest I start constructing and using a modular database?"

I would use one of the existing completely modular databases. I always use an ORM because I prefer to focus on solving a specific problem.

---

Edit

All SQL databases are "modular", if you define your schema wisely to keep the "modules" separate from each other. That's what a schema name is for -- to quality the tables within that schema. Nothing to it -- a standard part of SQL92.

For a "modular" application, just use a SQL database. Any SQL database.

You do have to define the schema, but this can be done "on the fly". Some ORM tools support defining a schema without you having to write a lot of DDL.

I like SQLAlchemy -- it handles access and schema definition. I use the Django ORM a lot. It does both schema definition and access, as well.

Answer 2

I personally think that such a plug-in architecture might be a good fit for a schemaless, document-oriented solution like CouchDB. You could assign each plugin its own namespace in the document store and allow the plug-in to store whatever documents it wants to based on whatever schema it may need.

However, that's not to say that document-oriented storage is a panacea that will solve all of your design issues, nor that it will entirely replace the RDBMS in your application. But it could certainly complement your plug-in architecture.

Answer 3

Here's a nice treatment of plugins in Python, no database thingies included: A Simple Plugin Framework.

Answer 4

Here's how I'd do it: Each plugin should create it's own private tables and perform operations by joining the private tables with the host tables.

Suppose your host application has a user table like: (users: id, name, password) and someone wants to write a plugin to store and display each user's blog address.

The naive approach will be to add a "blogaddress" column to the users table.

The modular approach will be to have a blog address table that can be joined with the host application's table at runtime. That way, when you want to uninstall the plugin, you can just delete the extra tables and things return to the way they were.

Relational databases are designed for the purpose of making joins easy anyway.