Tracking system and real time stats analysis in Python

views:

270

answers:

+3 Q:

Tracking system and real time stats analysis in Python

This question is related to an older question: http://stackoverflow.com/questions/2019096/mysql-tracking-system. In short: I have to implement a tracking system that will have high loads using Python. For the database part I've settled on mongoDB (which sounds like the right tool for this job). The development language will be Python.

I was thinking of using several instances of a CherryPy application behind nginx. The reasoning behind this is that I don't want to handle all the wsgi part myself, but on the other hand I don't need a full blown web framework since the app will be simple and there's no need for ORM.

My questions are:

Should I use the CherryPy builtin server or should I use Apache with modwsgi (or another server altogether)?
Does this sound like a reasonable approach (nginx, mongoDB)? If not what would you recommend?

Thank you in advance.

+1 A:

Have you checked out Graphite? It sounds like exactly the kind of thing that you need (looking at your other question) and was designed for application and server monitoring by the Orbitz team. It's extremely robust and easy to use for this sort of thing.

Here's the project site: http://graphite.wikidot.com/
With some screenshots: http://graphite.wikidot.com/screen-shots

Travis Bradshaw 2010-01-11 18:56:08

It looks great. I'll look over it (and I'll ask the other team members to look over it). Hopefully this will do the job. Thanks.

Alex 2010-01-11 19:35:40

+1 A:

Sounds like MongoDB will be a good fit for this - fast updates with advanced operators, and M/R for batch offline processing. I think CherryPy behind Nginx should work well too. If you go the mod_wsgi route just watch out for this issue.

mdirolf 2010-01-11 19:37:58

Thanks for the tip :).

Alex 2010-01-11 20:26:38

I'd argue that the Mongo documentation is mischaracterising the problem with using mod_wsgi. It says 'When running PyMongo with the C extension enabled it is possible to see strange failures when encoding due to the way mod_wsgi handles module reloading with multiple sub interpreters.' Any problem you see isn't because of how mod_wsgi handles module reloading, it is going to be because Mongo C extension hasn't been written properly to work in multiple sub interpreters at the same time. This will occur for any multi interpreter system, not just mod_wsgi. Mongo could fix the C extension.

Graham Dumpleton 2010-01-11 21:58:46

Graham - the docs are open, so feel free to fork and contribute changes that you think should be in there. You're right that we could change the C extension to handle that case, although I'd worry about the performance impact of any "fix". If you're interested in discussing further let's take it to http://jira.mongodb.org - thanks for the input!

mdirolf 2010-01-21 18:14:29

ansaurus

tags:

views:

answers:

Tracking system and real time stats analysis in Python

related questions