How to build a mini-network of small programs feeding each other data?

Question

I'm trying to simulate a real-time network where nodes are consumers and producers of different rates. How would I quickly implement a sample of this with Python? I was imagining that I'd write a simple program for each node, but I'm not sure how to connect them to each other.

Answer 1

Look into NetworkX a Python library for creating and manipulating networks. Edit: Attention, nearby/related to NetworkX, and also hosted at Los Alamos NL, is PyGraphviz, a utility to display graphs. Thank you to las3jrock to point out that I had the wrong link initialy.

You can either use NetworkX as-is, or get inspiration from this library (I wouldn't bother this library really has "all things graph" that seem to be needed for network simulation.) In any case, this type of graph creation and manipulation would allow you to represent (and grow / shrink / evolve) the network.

With a single centralized object/service based on a graph library as mentioned, you'd be left with creating one (or several) class to represent the behavior of the network nodes.

Then, depending on your needs, and if the network is relatively small, the network nodes could effectively be "let loose", and run inside of threads. Alternatively (and this is often easy to manage for simulations), there could be a centralized method which iterates through the nodes and invoke their "do_it" methods when appropriate. If the "refresh rate" of such a centralized controller is high enough, real-time clocks at the level of nodes can be used to determine when particular behaviors of the node should be triggered. This simiulation can be event driven, or simply polled (again if refresh period is low enough relative to the clocks' basic unit of time.)

The centralized "map" of the network would offer the various network-related primitives required by the network nodes to perform their "job" (whatever this may be). In other word, the nodes could inquire from the "map" the list of their direct neighbors, the directory of the complete network, the routes (and their cost) towards any given node etc, .

Alternatively, the structure of the network could be distributed over the nodes of the network itself, à la Internet. This decentralized approach has several advantages, but it implies adding logic and data to the nodes so they implement the equivalent of DNS and Routing. This approach also has the drawback of requiring that a fair amount of traffic between nodes be related to the discovery and maintenance of the topology of the network rather than to whatever semantics of communication/exchange the network is meant to emulate. In a nutshell, I wouldn't suggest using this decentralized approach, unless the simulation at hand was aimed at studying the very protocols of such distributed network management systems.

Edit:

The DES approach suggested in several replies certainly addresses the simulation part of the question. If the network part of question is important, implementing a virtual network based on a strong graph library will be an important part of the solution. Such an approach would expose more readily the dynamics of the system associated with the network topology.

Answer 2

Here's how to make a basic client/server program: http://wdvl.internet.com/Authoring/python/client/watts06152009.html

For the data to send from one to another, I'd recommend JSON, the most simple format ever. Check simplejson if you want to implement it on python: http://pypi.python.org/pypi/simplejson/

Answer 3

Inter-process communication is a generally hard thing to get right. You may want to consider if another approach would meet your needs, like a discrete events simulator.

In a DES, you don't actually do the work of each node, just simulate how long each node would take to get the work done. You can use a priority queue to keep track of the incoming and outgoing work, and instrument the system to keep track of the queue size globally, and for each node.

Perhaps if you provide more detail on what you're trying to accomplish, we can give more specific recommendations.

Edit: Python has a built-in priority queue in the heapq module, see http://docs.python.org/library/heapq.html.

Answer 4

Stick with traditional simulation structures, at least at first

Is it your goal to write an asynchronous system as an exercise? If so, then I guess you have to implement at least a multi-threaded if not multi-process or network system.

But if it's really a simulation, and what you want are the analysis results, implementing an actual distributed model would be a horribly complex approach that is likely to yield far less data than an abstract simulation would, that is, the simulation itself doesn't have to be a network of asynchronous actors communicating. That would just be a good way to make the problem so hard it won't get solved.

I say, stick with the traditional simulation architecture.

The Classic Discrete Event Simulation

The way this works is that as a central data structure you have a sorted collection of pending events. The events are naturally sorted by increasing time.

The program has a main loop, where it takes the next (i.e., the lowest-valued) event out of the collection, advances the simulation clock to the time of this event, and then calls any task-specific processing associated with the event.

But, you ask, what if something was supposed to happen in the time delta that the simulator just jumped across? Well, by definition, there was nothing. If an individual element of the simulation needed something to happen in that interval, it was responsible for allocating an event and inserting it into the (sorted) collection.

While there are many packages and programs out there that are geared to simulation, the guts of a simulation are not that hard, and it's perfectly reasonable to write it up from scratch in your favorite language.

Have fun!

Answer 5

Two things come to mind:

1- You could write one or more daemons with Twisted Python. (Be warned, twisted can get a little overwhelming , since its an event-driven async system ). Each daemon can bind to a port and make itself available to other daemons. Alternately, you could just run everything within one daemon, and just have each "process" that you script fire at a different interval.. and talk to one another through bound ports as well.

2- You could use a single event driven core -- there are a few --, and just fork a bunch of processes or threads for each task.

Answer 6

I like @DigitalRoss's and dcrosta's suggestions of a discrete-even simulation, and I'd like to point out that the sched module in the Python standard library is just what you need at the core of such a system (no need to rebuild the core on top of heapq, or otherwise). You just need to initialize a sched.scheduler instance, instead of the usual time.time and time.sleep, by passing it two callables that simulate the passage of time.

For example:

class FakeTime(object):
  def __init__(self, start=0.0):
    self.now = start
  def time(self):
    return self.now
  def sleep(self, delay):
    self.now += delay

mytimer = FakeTime()

and use s = sched.scheduler(mytimer.time, mytimer.sleep) to instantiate the scheduler.

Answer 7

Just use Stackless Python, create tasklets, connect them with channels, and everything will work. It is extremely simple.

Answer 8

This is something Stackless does very well.

Also, you could use generators / co-routines.

Interesting links:

http://www.python.org/dev/peps/pep-0342/

New users can only post 1 hyper link... so here is the other one

'/'.join(['http:/', 'us.pycon.org', '2009', 'tutorials', 'schedule', '1PM6/'])