I need a message/queuing solution for my web-based system

Question

Hi,

I am looking for a message/queuing solution for my web based system running on Ubuntu.

The system was built on the following technologies:

• Javascript (Extjs framework) - Frontend
• PHP
• Python (Daemon service which interacts with the encryption device)
• Python pyserial - (Serial port interactions)
• MySQL
• Linux - Ccustom bash scripts(to update DB/mail reports)

The system serves the following purpose:

• Capture client information on a distributed platform
• Encrypt/decrypt sensitive transactions using a Hardware device

System breakdown:

1. The user gains access to the system using a web browser
2. The user captures client information and on pressing "submit" button The data is sent to the encryption device and the system enters a wait state
3. The data is then encrypted on the device and sent back to the browser
4. The encrypted data is saved to the DB
5. System exits wait state and displays DONE message

Please note: I have already taken care of waiting/progress messages so lets omit that.

What I have done so far:

• I created a python daemon which monitors a DB view for any new requests
• The daemon service executes new requests on the device using pyserial and updates the requests table with a "response" ie. the encrypted content
• I created a polling service in PHP which frequently checks if there is a "response" in >the requests table for the specific request
• Created the Extjs frontend with appropriate wait/done status messages

The problem with the current setup:

• Concurreny - We expect > 20 users at any time submitting encryption/decryption requests using a database as a message/queuing solution is not scalable due to table locking and only 1 listening process which monitors for requests
• Daemon service - Relying on a daemon service is a bit risky and the DB overhead seems a bit high polling the view for new requests every second
• Development - It would simplify my development tasks by just sending requests to a encrypt/decrypt service instead of doing this whole process of inserting a request in the db,polling for the response and processing the request in the daemon service.

My Question:

What would be the ideal message/queening solution in this situation? Please take into >account my system exclusively runs on a Ubuntu O/S.

I have done a few Google services and came accross something called a "Stomp" server but it prove somewhat difficult to setup and lacked some documentation. Also I prefer the advice from individuals who have some experience in setting up something like this instead of some "how to" guide :)

Thank You for your time

Answer 1

I would do:

• The web component connects to the encryption daemon/service, sends the data and waits for the answer

The encryption daemon/service would:

• On startup, start a thread (SerialThread) of each of the available serial devices
• All 'serial threads' would then do a SerialQueue.get (blocking waiting for messages)
• A multi threaded TCP server, check ThreadingMixIn from http://docs.python.org/library/socketserver.html
• The TCP Server threads would receive the plain data and put it on the SerialQueue
• A random SerialThread (Python's Queue class manages the multi thread required locking for you) would receive the request, encrypt and return the encrypted data to the TCP Server thread
• The TCP Server thread would write the data back to the web component

I am using this logic on a project, you can check the source at http://bazaar.launchpad.net/~mirror-selector-devs/mirror-selector/devel/files/head:/mirrorselector/, on my case the input is an URL, the processing is to scan for an available mirror, the output is a mirror url.

Answer 2

I believe the popular RabbitMQ implementation of AMQP offers a PHP extension (here) and you can definitely access AMQP in Python, e.g. via Qpid. RabbitMQ is also easy to install on Ubuntu (or Debian), see e.g. here.

Whether via RabbitMQ or otherwise, adopting an open messaging and queueing protocol such as AMQP has obvious and definite advantages in comparison to more "closed" solutions (even if technically open source, such solutions just won't offer as many implementations, and therefore flexibility, as a widely adopted open, standard protocol).