Receiving the Same MSMQ Message Twice?

Question

I have an MSMQ-based system with three layers that communicate with each other. For the sake of simplicity, I'll refer to them as Layer 1, 2, and 3. They sit like this:

Layer 1 <-> Layer 2 <-> Layer 3

So Layer 1 talks only to Layer 2, Layer 3 talks only to Layer 2, and Layer 2 talks to both others. I have four queues for this,

Layer1_in
Layer1_out
Layer3_out
Layer3_in

And the layers communicate over this infrastructure:

Layer 1 -> Layer1_out -> Layer 2
Layer 1 <- Layer1_in <- Layer 2
Layer 3 -> Layer3_out -> Layer 2
Layer 3 <- Layer3_in <- Layer 2

(sorry if this is more exhaustive than necessary)

Anyhow, a message gets passed from Layer 1 to Layer 2, some processing takes place, then another (related) message is sent to Layer 3, and vice versa. The issue I'm having is that occasionally I have two messages sent from Layer 1 to Layer 2, but instead of receiving both messages in order it's receiving the first message twice. I'm using BeginReceive on Layer1_out to asynchronously receive the message. When that completes I process the received message and call BeginReceive again to get the next message.

To track this down I implemented a message counter on the sending side and I write it to a text file. I'm using the Extension property to store a string representation of this message number so that I can then retrieve the message number on the processing side. When I recieve a message, I take this number and write it to a different file. This SHOULD produce two files that have the same contents, but instead I'll see things like

00000000000000000214
00000000000000000215
00000000000000000215 <- this is bad!
00000000000000000217
00000000000000000218
00000000000000000219

In the receipt log, indicating that message 215 was processed twice and 216 didn't come through. For my purposes it doesn't affect anything for me to process the same message twice (hence why this has gone unnoticed for some time), but missing a message altogether is a big problem.

Any thoughts?

Answer 1

Problem solved...figures that I would dole out multithreading advice one day then get bitten by a violation myself the next!

The culprit was this line:

receiveResult = <queueName>.BeginReceive()

When I had another message already waiting in the queue, my ReceiveCompleted event was firing on a different thread and getting to my EndReceive(receiveResult) call before the return value from BeginReceive was actually written to the receiveResult variable. As a result, I was calling EndReceive with the same value and getting the same message again! Locking the whole event handler around a sync object took care of the issue.