Guaranteeing every line is received in full from multiple PIPEs (STDOUTs)

Question

Hello,

I asked the other day if data integrity (of flushed data) is kept even when there are more than one PIPEs streaming into localhost's STDIN. The answer is NO if the data flushed is large. http://stackoverflow.com/questions/3445047/data-integrity-question-when-collecting-stdouts-from-multiple-remote-hosts-over-s

But I would like to guarantee every line flushed on each end is passed to the single STDIN in full and won't be mixed up with data from other pipes. Is there any way to do so? How can that be done?

(Note that it can be done if I create multiple STDINs locally. But it is more convenient if I can process line streams through a single STDIN. So my question focuses on the case when there is only one STDIN at localhost with multiple (STDOUT) PIPEs into it.)

Answer 1

This can be done via a congestion-backoff system like that used in Ethernet.

First, assign each pipe a unique delimiter. This delimiter cannot appear unescaped in the contents of any pipe. Now, use the following pseudocode:

• Check for other process' delimiter; while an odd number of a single other process' delimiters is present, wait.
• Write delimiter character.
• Check if another process has also written an unmatched delimiter. If so, back off a random (increasing) amount and return to first step.
• Write data.
• Write delimiter character.

This will ensure that, although you will have some junk, every whole message will eventually get through.