TCP multicast and multithreading

views:

176

answers:

+1 Q:

TCP multicast and multithreading

I need to come up with clients that can multicast to other clients reliably. That implies I'll be using TCP to connect reliably between clients within a multicast group. Doesn't that come up to n^2 number of connections? That seems a little silly to me. Wouldn't/shouldn't there be a way to more easily multicast with reliability?

EDIT: UNIX/C

EDIT: i didn't clarify how multithreading comes into play. but if i was to open up n^2 connections, i figured, i'd be multithreading and that's even more complication than i would want.

Sure there is a more efficient way - you stay using UDB, and reimplement reliable sending yourself. Not trivial, though. But at least you only need to keep sent packets ONCE on the sending site.

TomTom 2010-03-16 09:11:39

Those who do not use TCP, are doomed to re-implement it, to paraphrase someone. Not the best of all possible worlds.

Andrew B 2010-03-16 16:06:36

Pretty ignorant answer. TCP is point to point, he is talking about broadcast. Reimplementing part of TCP is the only solution if you do NOT want the traffic to be distributed X times (with x being the number of listeners). Reality does not bend to Andrew B's funny ideas.

TomTom 2010-03-16 20:25:12

+2 A:

multicast and TCP are mutually exclusive.

Implementing reliable delivery over UDP is nuts. Nobody does this since 1980s and it is impossible to do as good as any cheap TCP stack, in terms of performance and BW overhead. Correction: sometimes it is done manually, but only over exotic transports, such as extremely long or narrow pipes.

N^2 connections is not very silly. A connection with 1Hz keepalives does not cost much. What costs is the traffic. This is what your design needs to focus on.

Pavel Radzivilovsky 2010-03-16 09:20:32

the first part is correct - but for massive data scenarios in financial services UDP with sequence numbers is still the way to go - even the latest implementation from exchange use this.

weismat 2010-03-16 09:31:11

I disagree. From extensive experience in examining numerous financial applications (consulting TASE, others) any team that went for UDP was sorry for it. They would provide irrelevant excuses like "system overhead for the target application"

Pavel Radzivilovsky 2010-03-16 10:41:42

What makes you say no body implements reliable multicast? You're wrong --> http://en.wikipedia.org/wiki/Reliable_multicast

Jorge Córdoba 2010-03-17 23:00:41

@Pavel Radzivilovsky I wouldn't say implementing reliable UDP is 'nuts' - though I'd agree it's not at all straightforward. We do use reliable UDP quite a bit in networked multi-player games code though I will admit I wouldn't fancy implementing it myself :)

zebrabox 2010-03-17 23:04:45

@zebrabox: there's more to nuts than difficulty. I don't argue about possibility of doing reliable UDP. Yet, any implementation I have seen missed away so many of the important features of modern TCP stacks, I can't even start enumerating them. I would scrap all these implementations. They are no more than a second year student's NIH.

Pavel Radzivilovsky 2010-03-18 09:18:40

@Pavel; surely the point of these various reliable UDP systems is that they miss out various aspects of TCP. It's a trade off. Not everyone values the all of the features of TCP equally. That said, it's easy to make a mess of it...

Len Holgate 2010-03-19 21:12:30

TCP may be not good for someone? Yes, strictly speaking maybe it can sometimes be true. In a real life situation? Never.

Pavel Radzivilovsky 2010-03-20 20:17:28

@Pavel Radziviovsky - Care to outline why TCP is good in all circumstances in 'real life'? TCP works best when the packets are reliant on each other - then the error checking comes into it's own but this means high transactional latency on packet loss. If your packets are not reliant on each other than reliable UDP is a decent alternative - though I agree it's more specialised and for the vast majority of cases TCP is a better choice

zebrabox 2010-03-21 01:48:51

there's no argument about situations when order is important. My claim is about the other situations. show me when you deliberately want misordered packets for latency in unstable traffic. These cases are extremely rare. But yes, I admit, they exist.

Pavel Radzivilovsky 2010-03-21 20:11:24

+2 A:

Depending on your target platform....

You could take a look at Pragmatic General Multicast. This is, as I understand, what Microsoft MSMQ and Tibco Rendezvous use and it can be accessed via Winsock (see: http://msdn.microsoft.com/en-us/library/ms740125(VS.85).aspx).

Len Holgate 2010-03-16 11:46:38

PGM is an option, as mentioned above. The issue we had with it is that if a client can't keep up with reading incoming data, it get's disconnected.

Since we never could reliable guarantee 'fast enough' clients, we've implemented our own reliability protocol on top of UDP multicast. The implementation is not fully generic when it comes to dynamic connects/disconnects, but it might work for you. You can find the implementation here:

http://www.equalizergraphics.com/cgi-bin/viewvc.cgi/trunk/src/lib/net/rspConnection.h?view=markup http://www.equalizergraphics.com/cgi-bin/viewvc.cgi/trunk/src/lib/net/rspConnection.cpp?view=markup

eile 2010-03-16 13:40:10

FYI: The OpenPGM implementation of the PGM protocol allows you to completely customize the protocol parameters such as never disconnecting on unrecoverable data loss.

Steve-o 2010-03-19 10:23:37

Just a thought, but does your work need to be done with a networking protocol. You might also look at implementing this with a message service, with a publish-subscribe based model.

If you do need networking, then you're going to have to deal with either lots of connections, or ensuring delivery yourself. Make very sure of your requirements before going down that road.

Andrew B 2010-03-16 16:09:06

+1 A:

You need to take a look at 0MQ which is a high speed Messaging System which has as one of it's abilities reliable multicast using the Pragmatic General Multicast (PGM) using OpenPGM.

There was an article on it recently in lwn.net:

0MQ: A new approach to messaging

Robert S. Barnes 2010-03-17 22:50:53

The choice between using OpenPGM directly or 0MQ is determined by how you want to handle threading as 0MQ provides a thread pool for IO handling whilst OpenPGM has no internal threads at all.

Steve-o 2010-03-19 10:21:49

There are several reliable multicast solutions.

DDS (Data distribution service)
Norm Protocol
PGM

I've tried the first two ones.

Norm is simple, works like standard udp multicast but incorporates nacks... excelent if you do not need more. There are some implementations that aslo support bandwidth adaptation and other improvements.

DDS is a step forward. It's really great (I know the RTI implementation and it works great) and has a lot of capabilities as well as a very good though design. It's based on reliable and fault tolerancy and there's an open implementation.

By the way, at least DDS and NORM do not require n^2 connections. They work like multicast udp.

Jorge Córdoba 2010-03-17 22:59:16

Thanks, i'm not quite using DDS but using their model to solve my problem.

Fantastic Fourier 2010-03-17 23:57:32

ansaurus

tags:

views:

answers:

TCP multicast and multithreading

related questions