How do I scp the (huge) output of a command directly to a remote machine?

Question

Note that I can't first store the file locally -- it's too big.

This (obnoxious) page (scroll all the way to the bottom) seems to give an answer but I'm having trouble disentangling the part that's specific to tape drives:

http://webcache.googleusercontent.com/search?q=cache:lhmh960w2KQJ:www.experts-exchange.com/OS/Unix/SCO_Unix/Q_24249634.html+scp+redirect&cd=3&hl=en&ct=clnk&gl=us

To make this more concrete, here's how you would think it might work:

On local machine:

% echo "pretend this string is a huge amt of data" | scp - remote.com:big.txt

(That's using the convention -- which scp does not in fact support -- of substituting a dash for the source file to tell it to get it from stdin instead.)

Answer 1

Use nc (Net Cat), which doesn't need to save the file locally.

Answer 2

Use a FIFO pipe:

mknod mypipe p
scp mypipe destination &
ls > mypipe

Answer 3

Using ssh:

echo "pretend this is a huge amt of data" | ssh [email protected] 'cat > big.txt'

Answer 4

You can pipe into ssh and run a remote command. In this case, the remote command is cat > big.txt which will copy stdin into the big.txt file.

echo "Lots of data" | ssh [email protected] 'cat > big.txt'

It's easy and straightforward, as long as you can use ssh to connect to the remote end.

You can also use nc (NetCat) to transfer the data. On the receiving machine (e.g., host.example.com):

nc -l 1234 > big.txt

This will set up nc to listen to port 1234 and copy anything sent to that port to the big.txt file. Then, on the sending machine:

echo "Lots of data" | nc host.example.com 1234

This command will tell nc on the sending side to connect to port 1234 on the receiver and copy the data from stdin across the network.

However, the nc solution has a few downsides:

• There's no authentication; anyone could connect to port 1234 and send data to the file.
• The data is not encrypted, as it would be with ssh.
• If either machine is behind a firewall, the chosen port would have to be open to allow the connection through and routed properly, especially on the receiving end.
• Both ends have to be set up independently and simultaneously. With the ssh solution, you can initiate the transfer from just one of the endpoints.