Django database scalability

Question

Hi ALL:

We have a new django powered project which have a potential heavy-traffic characteristic(means a heavy db interaction). So we need to consider the database scalability in advance. With some researches, the following questions are still not clear to us:

1. coarse-grained: how to specify one db table(a django model) to a specific db(maybe in another server)?
2. fine-grained: how to specify a group of table rows to a specific db(so-called sharding, also can in another db server)?
3. how to specify write and read to different db?(which will be helpful for future mysql master/slave replication)

We are finding the solution with:

1. be transparent to application program(means we don't need to have additional codes in views.py)
2. should be in ORM level(means only needs to specify in models.py)
3. compatible with the current(or future) django release(to keep a minimal change for future's upgrading of django)

I'm still doing the research. And will share in this thread later if I've got some fruits.

Hope anyone with the experience can answer. Thanks.

Answer 1

hi there, here for some reason we r using django with sqlalchemy. maybe combination of django and sqlalchemy also works for your needs.

Answer 2

1. There is the GSoC project by Alex Gaynor that in future will allow to use multiple databases in one Django project. But now there is no cross-RDBMS working solution.

2. There is no solution right now too.

3. And again - there is no cross-RDBMS solution. But if you are using MySQL you can try excellent third-party Django application called - mysql_replicated. It allows to setup master-slave replication scenario easily.

Answer 3

Don't forget about caching either. Using memcached to relieve your DB of load is key to building a high performance site.

As alex said, django-core doesn't support your specific requests for those features, though they are definitely on the todo list.

If you don't do this in the application layer, you're basically asking for performance trouble. There aren't any really good open source automation layers for this sort of task, since it tends to break SQL axioms. If you're really concerned about it, you should be coding the entire application for it, not simply hoping that your ORM will take care of it.