Add last n lines of files to tar/zip

Question

I need to regularly send a collection of log files that can grow quite large, so I would like to only send the last n lines of the each of the files.

for example:

/usr/local/data_store1/file.txt (500 lines)
/usr/local/data_store2/file.txt (800 lines)

Given a file with a list of needed files named files.txt, I would like to create an archive (tar or zip) with the last 100 lines of each of those files.

I can do this by creating a separate directory structure with the tail-ed files, but that seems like a waste of resources when there's probably some piping magic that can happen to accomplish it. Full directory structure also must be preserved since files can have the same names in different directories.

I would like the solution to be a shell script if possible, but perl (without added modules) is also acceptable (this is for Solairs machines that don't have ruby/python/etc.. installed on them.)

Answer 1

You could try

tail -n 10 your_file.txt | while read line; do zip /tmp/a.zip $line; done

where a.zip is the zip file and 10 is n or

tail -n 10 your_file.txt | xargs tar -czvf test.tar.gz --

for tar.gz

Answer 2

Why not put your log files in SCM?

Your receiver creates a repository on his machine from where he retrieves the files by checking them out.

You send the files just by commiting them. Only the diff will be transmitted.

Answer 3

There is no piping magic for that, you will have to create the folder structure you want and zip that.

mkdir tmp
for i in /usr/local/*/file.txt; do
    mkdir -p "`dirname tmp/${i:1}`"
    tail -n 100 "$i" > "tmp/${i:1}"
done
zip -r zipfile tmp/*

Answer 4

Use logrotate.

Have a look inside /etc/logrotate.d for examples.

Answer 5

You are focusing in an specific implementation instead of looking at the bigger picture.

If the final goal is to have an exact copy of the files on the target machine while minimizing the amount of data transfered, what you should use is rsync, which automatically sends only the parts of the files that have changed and also can automatically compress while sending and decompress while receiving.

Running rsync doesn't need any more daemons on the target machine that the standard sshd one, and to setup automatic transfers without passwords you just need to use public key authentication.