Backend processing for Django

Question

I'm working on a turn-based web game that will perform all world updates (player orders, physics, scripted events, etc.) on the server. For now, I could simply update the world in a web request callback. Unfortunately, that naive approach is not at all scalable. I don't want to bog down my web server when I start running many concurrent games.

So what is the best way to separate the load from the web server, ideally in a way that could even be run on a separate machine?

• A simple python module with infinite loop?
• A distributed task in something like Celery?
• Some sort of cross-platform Cron scheduler?
• Some other fancy Django feature or third-party library that I don't know about?

I also want to minimize code duplication by using the same model layer. That probably means my service would need access to the Django model code, so that definitely determines how I architect the service.

Answer 1

I'd just write the backend to just use the Django database interface (look at the setup code in your manage.py), spawn it as its own process, and interface to it with Protocol Buffers. That route should move to a separate machine with little work. MPI may be an option, too.

Pipes, FIFOs, and most other IPC requires both processes to be on the same box.

Though I have to point out a flaw in your premise:

Unfortunately, that naive approach is not at all scalable. I don't want to bog down my web server when I start running many concurrent games.

If you run concurrent games, so long as you keep all the parts for a given game on the same server, this is a non-issue, unless there's a common resource needed by all games. Then the real issue becomes load balancing across the servers.

Answer 2

I think Celery, which you mention in your question, is the way to go here. It will interface nicely with the rest of your setup, support your eventual aim of separating out the systems, and is compatible with Django.