rsync hang with many remote files specified

Question

Using the following rsync command will let you copy numerous remote files to your local machine, when the remote machine is using rsyncd:

rsync -R hostname::'module/path1/file1 module/path2/file2 module/path3/file3 module/path4/file4' /local/path

If the number of files is too great, however, rsync hangs, not creating the files in the target location. The limit in the environment I'm testing on is 22. With 22 files it finishes in a few seconds; with 23 or more files, it hangs indefinitely.

Can anyone provide a reason why this might happen? Here's some things I've checked:

• Neither machine is running out of disk space or memory.
• All the files can be transferred, as long as I don't do more than 22 at a time. (In other words, it's not something special about the specific files at position 23 and above.)
• There are no permissions issues; I have the ability to read all files on the remote machine, and can write to the target destination. (Also, as I noted in the previous item, all of the ~50 files in my overall list can be transferred as long as I don't do more than 22 at a time.)
• Nothing changes if I use the -n (dry run) option.

Answer 1

strace the client and server, see who's causing the problem. It smells like a bug of some sort in rsync.

Answer 2

Okay, after further testing, it appears to have to do with argument length. I haven't worked out the exact length, but I tried a different set of 25 files, with shorter path/filenames, and it worked normally. So I guess I just have to limit the length of the command. Arse. :\

Answer 3

I always recommend scp. Very reliable, copies via ssh.

Answer 4

Let's assuming this is a bug rather than an expected behavior.

Since Rsync is an open source project. you could download the source, find the bug and upload a fix for everyone. You should check if you're using the latest version first.

Answer 5

There's no explanation of why you're specifying individual files. The simplest approach is to rsync an entire directory recursively, and use either the --include or --exclude flags with the appropriate patterns you want to match on if you need more control.

If your usage really does require explicitly specifying every single file to be backed up, then use a separate text file that lists those files and call rsync with the --files-from flag.

I've used the --files-from option. It works well. rsync is very solid - but the many configuration options can make it confusing to learn. For example, there's a --cvs-exclude flag to exclude all the metadata files from backup not only for CVS but also for SVN. Extremely useful, but definitely not easily discoverable.

All of these options are detailed in the rsync documentation here.