Read data from pipe and write to standard out with a delay in between. Must handle binary files too.

Question

I have been trying for about an hour now to find an elegant solution to this problem. My goal is basically to write a bandwidth control pipe command which I could re-use in various situations (not just for network transfers, I know about scp -l 1234). What I would like to do is:

1. Delay for X seconds.
2. Read Y amount (or less than Y if there isn't enough) data from pipe.
3. Write the read data to standard output.

Where:

• X could be 1..n.
• Y could be 1 Byte up to some high value.

My problem is:

• It must support binary data which Bash can't handle well.

Roads I've taken or at least thought of:

• Using a while read data construct, it filters all white characters in the encoding your using.
• Using dd bs=1 count=1 and looping. dd doesn't seem to have different exit codes for when there were something in if and not. Which makes it harder to know when to stop looping. This method should work if I redirect standard error to a temporary file, read it to check if something was transfered (as it's in the statistics printed on stderr) and repeat. But I suspect that it's extremely slow if used on large amounts of data and if it's possible I'd like to skip creating any temporary files.

Any ideas or suggestions on how to solve this as cleanly as possible using Bash?

Answer 1

Do you have to do it in bash? Can you just use an existing program such as cstream?

cstream meets your goal of a bandwidth controlled pipe command, but doesn't necessarily meet your other criteria with regard to your specific algorithm or implementation language.

Answer 2

What about using head -c ?

cat /dev/zero | head -c 10 > test.out

Gives you a nice 10 bytes file.

Answer 3

It's not much elegant but you can use some redirection trick to catch the number of bytes copied by dd and then use it as the exit condition for a while loop:

while [ -z "$byte_copied" ] || [ "$byte_copied" -ne 0 ]; do 
    sleep $X; 
    byte_copied=$(dd bs=$Y count=1 2>&1 >&4 | awk '$2 == "byte"{print $1}'); 
done 4>&1

However, if your intent is to limit the transfer throughput, I suggest you to use pv.