What are good specs/libraries for closed network communication in python?

Question

The situation is that I have a small datacenter, with each server running python instances. It's not your usual distributed worker setup, as each server has a specific role with an appropriate long-running process.

I'm looking for good ways to implement the the cross-server communication. REST seems like overkill. XML-RPC seems nice, but I haven't played with it yet. What other libraries should I be looking at to get this done?

Requirements:

Computation servers crunch numbers in the background. Other servers would like to occasionally ask them for values, based upon their calculation sets. I know this seems pretty well aligned with a REST mentality, but I'm curious about other options.

Answer 1

Twisted's Perspective Broker is an extremely easy to use and robust mechanism for cross-server communication. It's definitely worth a look.

Answer 2

It wasn't obvious from your question but if getting answers back synchronously doesn't matter to you (i.e., you are just asking for work to be performed) you might want to consider just using a job queue. It's generally the easiest way to communicate between hosts. If you don't mind depending on AWS using SQS is super simple. If you can't depend on AWS then you might want to try something like RabbitMQ. Many times problems that we think need to be communicated synchronously are really just queues in disguise.