How to use message queues when working with parrallel processes in php?

Question

Hi,

I'm trying to implement a program where a parent process opens multiple child processes (usually around 40) that run in parallel to each other.

Now, I want to be able to get the child processes talking to the parent process to extract some information that only the parent process has. My plan was to use message queues in php but I'm running into some issues. Here is my basic design:

1) Parent process starts 2) Parent process opens multiple child processes 3) Each child process sends multiple messages to the parent process. For example a particular child can send anywhere from 20 to 100 messages to the parent. 4) Parent receives each messages and sends a 'return' message to the child, for each message that the parent receives 5) Child receives the replies from the parent and performs some action

Unfortunately, I haven't been able to get this to work. I've tried many different things such as have a separate queue for each child, which gave me a "Warning: msg_get_queue() [function.msg-get-queue]: failed for key No space left on device in ..." even though I was deleting the queues after I was done with them. Basically, the system opened around 10-15 queues but after that it couldn't handle any more queues it seems and started to throw that error.

I also tried giving each child process its own desiredmsgtype (see the msg_get_queue() function) when sending a message and each parent message its own unique desiredmsgtype when sending a reply but that didn't work either. Either some of my messages were being lost or I started to get the error "msgsnd: Resource temporarily unavailable" when the children tried to send their messages to the parent (Note that some of the messages from the children did go through).

So for example:

child code:

$child_msg = (object)array('unique_msg_id' => $unique_msg_id);

if (!msg_send ($ipc_msg_queue, $child_msg_id, $child_msg, true, true, $msg_err))
{
    echo "Msg not sent because $msg_err\n";
}

$parent_msg_id = $unique_msg_id;

if (msg_receive ($ipc_msg_queue, $parent_msg_id, $msg_type, 16384, $msg, true, 0, $msg_error))
{
    echo "Msg from parent with id: $parent_msg_id";
}

parent code:

while each child process is running:

    while (msg_receive ($ipc_msg_queue, $child_msg_id, $msg_type, 16384, $msg, true, 0, $msg_error))
    {
        echo "Msg from child with id: $child_msg_id";
    }

    $parent_msg_id = $msg->unique_msg_id;
    $parent_msg = 'perform some action';

    if (!msg_send ($ipc_msg_queue, $parent_msg_id, $parent_msg, true, true, $msg_err))
    {
        echo "Msg not sent because $msg_err\n";
    }

I'm on a Linux box and I looked at /proc/sys/fs/mqueue/msg_max to see the limit for the maximum messages allowed in a queue but found that that file does not even exist. Could that be a reason for my problems?

So, I want to know:

How you would go about solving this problem by using message queues in php?

Why do you think I might be getting the errors that I have identified above?

Thanks in advance

Answer 1

Consider using an entirely different message queue system, like Gearman. It's meant as a distributed work queue. It has an extensive PHP extension. Check out the examples and see if it meets your needs. It should be a great replacement for what you're trying (unsuccessfully) to do, given your description.

Answer 2

If you're fork()ing new children, you're copying the parent's memory for each child. If you create a message queue for each child as you create these forks, each child gets a copy of the message queue for all children before it.

I'm not sure if you've considered this, though you probably have.

Regards, Paul