I want to push data to the browser over HTTP without killing my django/python application.
I decided to use a comet server, to proxy requests between my application and the client (though I still haven't really figured it out properly).
I've looked into the following engines: orbited cometd ejabberd jetty
Has anyone had any experience working with these servers and deploying them? Any insight and links regarding the topics would be great. Thank you.
I need to do something very similar. I found this, but haven't had the time to look at it properly yet:
django_evserver is simple http server for Django applications. It's based on libevent library. The main advantage of django_evserver is that it provides methods of preempting django views. You can render a site in several chunks, each of them can be handled by different django view.
Using this idea it's possible to use django_evserver as comet server for django applications.
I would recommend looking into Twisted, their twisted.web server, and the comet work done on top of it at Divmod. They can handle far more concurrent connections than traditional thread or process based servers, which is exactly what you need for something like this. And, yes, I've architected systems using Twisted for COMET stuff, while using other things for the more front-facing web applications beside it. It works out well with each part doing what it does best.
If you can run Java I would recommend StreamHub Comet Server.
Firstly, regarding your need not to 'kill' your existing application, the JavaScript include has a really low footprint at less than 10K. I use it for pushing user updates and chat on the social networking site I'm building. I tested with a 1000+ hits a day and there was no noticeable effect on the CPU.
Secondly, on deploying, I followed some of the examples and was up and running really quickly compared to banging my head against a wall with CometD. There is a good Comet Hello World getting started guide and a Google Group if you get stuck on anything.
One option is Netty, client-server socket framework based on Java NIO from JBoss. For a comparison and discussion see here. It reportedly handles 100000 simultaneous open connections on a quad-core server.
If you're running IIS, you can check out WebSync (http://www.frozenmountain.com/websync), a standards-compliant (bayeux) comet server and client for .NET/IIS. If you don't want the additional load, the On-Demand version is a SaaS option that offloads the heavy lifting.
If you're looking to combine Django with a Comet server (Orbited), check this project I have going to integrate Django and Orbited in as "clean" and "real-world" as possible here: http://github.com/clemesha/hotdot
The project addresses "real-world" problems like security and logging/filtering/modifying the in-transit Comet messages, etc - but is still a work in progress.