How can I display rsync progress from a log file without the carriage returns?

Question

I have a log file from rsync command with progress on. When running this progress updates the display information on the same line. When I capture the output from this command I get a file that displays normally with cat on a terminal (all the backspaces and re-edits are replayed) but I want to be able to use grep on the file and process it so I see all the backspace edit commands. How can I process the file to remove all the progress updates and just get a file that has the final edits?

Answer 1

Here is an answer I suspect will work, but am not positive will work.

First, I'm guessing the progress indicators are important for you to keep for various reasons, because the simplest solution is to just turn them off. Secondly, I'm assuming there is no way to tell rsync to write out a log with everything but the progress indicators to a separate file from what it shows on the screen. Lastly, I'm assuming that you do not want any of the progress indicator stuff at all in the log file you later process with other tools.

Given these two assumptions, you can strip out the progress indicators by basically looking through each line for \b or \r characters and tossing all lines that have those characters. This can be as simple as a grep command line that looks like this:

grep -ve "$(echo -ne '[\b\r]')" logfile

This presupposes you have a shell that supports "$(...)" style argument substitution and an echo command that supports the -e arguments.

Answer 2

If you want to see what your captured file looks like without the progress lines overwriting the previous ones:

sed 's/\r/\n/g' capture-file

which might yield something like this:

created directory /some/dest/dir
filename

       32768   0%    0.00kB/s    0:00:00
     6717440  21%    6.38MB/s    0:00:03
    13139968  41%    6.25MB/s    0:00:02
    19791872  62%    6.28MB/s    0:00:01
    26214400  82%    6.24MB/s    0:00:00
    31784420 100%    6.25MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
total size is 31784420  speedup is 1.00

If you want to see just the final step of the progress message and eliminate the previous ones:

sed 's/.*\r/\n/g' capture-file

Which might look like this:

created directory /some/dest/dir
filename

    31784420 100%    6.25MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
total size is 31784420  speedup is 1.00

You can run rsync with the --log-file=name option to capture log information in a file. Replace "name" with the name that you want. You can control the information that gets logged using the log-file-format option (see the log format section in man rsyncd.conf for details).

On my system the default rsync log file looks like this:

2009/11/01 17:19:20 [23802] building file list
2009/11/01 17:19:20 [23802] created directory /some/dest/dir
2009/11/01 17:19:25 [23802] <f+++++++++ filename
2009/11/01 17:19:25 [23802] sent 31788388 bytes  received 31 bytes  3346149.37 bytes/sec
2009/11/01 17:19:25 [23802] total size is 31784420  speedup is 1.00