Here's the jist of some code I am currently using in a production app that works (we've got a bit extra in there to handle the case where the client are server apps are running on a standalone installation). It's job is to receive udp notifications that messages are ready for processing. As mentioned by Adam Alexander your only problem is that you need to use IPAddress.Any, instead of IPAddress.Broadcast. You would only use IPAddress.Broadcast when you wanted to Send a broadcast UDP packet.
Set up the udp client
this.broadcastAddress = new IPEndPoint(IPAddress.Any, 1234);
this.udpClient = new UdpClient();
this.udpClient.Client.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
this.udpClient.ExclusiveAddressUse = false; // only if you want to send/receive on same machine.
And to trigger the start of an async receive using a callback.
this.udpClient.Client.Bind(this.broadcastAddress);
this.udpClient.BeginReceive(new AsyncCallback(this.ReceiveCallback), null);
Hopefully this helps, you should be able to adapt it to working synchronously without too much issue. Very similar to what you are doing. If you're still getting the error after this then something else must be using the port that you are trying to listen on.
So, to clarify.
IPAddress.Any = Used to receive. I want to listen for a packet arriving on any IP Address.
IPAddress.Broadcast = Used to send. I want to send a packet to anyone who is listening.