Adding methods to a simple RPC server in a clean and separated way

Question

Hello:

I created a simple RPC server to perform certain tasks common to our teams, but which are called from different networks. The server looks like this (I don't include error handling for brevity):

from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
import json

class MyProtocol(Protocol):
    def dataReceived(self, data):
        req = json.loads(data) # create a dictionary from JSON string
        method = getattr(self, req['method']) # get the method
        method(req['params']) # call the method

    def add(self, params):
        result = {} # initialize a dictionary to convert later to JSON
        result['result'] = sum(params) 
        result['error'] = None 
        result['id'] = 1
        self.transport.write(json.dumps(result)) # return a JSON string
        self.transport.loseConnection() # close connection

factory = Factory()
factory.protocol = MyProtocol
reactor.listenTCP(8080, factory)
reactor.run()

This is very simple: the server receives a JSON RPC request from the client, looks for the method, and calls the method passing the parameters. The method itself is the one returning the JSON RPC response. For the less familiar, a JSON RPC looks approximately like this:

request:
{"method":"my_method", "params":[1,2,3], "id":"my_id"}
response:
{"result":"my_result", "error":null, "id":"my_id"}

The RPC server as I have it serves my current purposes very well (as you can imagine, my task is very simple). But I will need to continue adding methods as the complexity of the task increases.

I don't want to open the main file and add yet another def method3(...) and, two weeks later, add def method4(...) and so forth; the code would grow too quickly and the maintenance would be harder and harder.

So, my question is: how can I create an architecture that allows me to register methods into the Server. A bonus would be to have a separate folder holding one file per method, so that they can easily be shared and maintained. This "architecture" would also allow me to defer maintenance of some methods to someone else, regardless of their understanding of Twisted.

I don't care if I need to restart the server every time a new method is registered, but an obvious plus would be if I don't have too :).

Thanks.

Answer 1

A bit of a largish order ;) but here's some initial steps for you (very heavily mocked-up, twisted specifics ommited in the examples):

# your twisted imports...
import json

class MyProtocol(object): # Would be Protocol instead of object in real code

    def dataReceived(self, data):
        req = json.loads(data) # create a dictionary from JSON string
        modname, funcname = req['method'].split('.')
        m = __import__(modname)
        method = getattr(m, funcname) # get the method
        method(self, req['params']) # call the method

Assuming you try it out as if we executed this:

mp = MyProtocol()
mp.dataReceived('{"method":"somemod.add", "params":[1,2,3]}')

You wold have a module somemod.py in the same directory as the example with the following contents (mirroring your example method .add() above):

import json

def add(proto, params):
    result = {} # initialize a dictionary to convert later to JSON
    result['result'] = sum(params)
    result['error'] = None
    result['id'] = 1
    proto.transport.write(json.dumps(result)) # return a JSON string
    proto.transport.loseConnection() # close connection

This allows you to have one module per method served. The method(.. call above will always pass your MyProtocol instance to the serving callable. (If you really want instance methods, here's instructions on how to add methods using python: http://irrepupavel.com/documents/python/instancemethod/ )

You will need a lot of error handling added. For example you need a lot of error checking at the split() call on line 2 of dataReceived().

With this you can have separate files with one function in them for every method you need to support. By no means a complete example but it might get you going, since what your'e looking for is quite complex.

For a more formal registering, I'd recommend a dict in MyProtocol with names of methods that you support, along the lines of:

# in MyProtocol's __init__() method:
self.methods = {}

And a register method..

def register(self, name, callable):
    self.methods[name] = callable

..modify dataReceived()..

def dataReceived(self, data):
    # ...
    modname, funcname = self.methods.get(req['method'], False)
    # ..continue along the lines of the dataReceived() method above

Quick summary of a too long post: the __import__ function ( http://docs.python.org/library/functions.html ) will most certainly be a key part of your solution.