efficient way to merge a head file and part of a tail file and then to stdout

Question

Actually I am in great agony recoverying a corrupt gzip file, possibly due to interrupted ftp transfer and then resume. After googling I found Recovering a damaged .gz file and am trying as it reads.

What I'm doing now is merging a gzip header file with some last part of the damaged file varying the size of the last part. Then I test the merged file as a gunzip input if it can produce a meaningful result. I wrote a script and the following is the best what I can to redirect the merge to gunzip. What is more efficient way to redirect file content to gzip? I guess you should not create any file. ($i is a size variable)

cat head > x.gz; tail -c $i tail >> x.gz; gzip -t x.gz 2>&1 1>/dev/null

Answer 1

Same but without a temp file:

for ((i=0; i<$TAIL_FILE_SIZE; $i++)); do
  ( cat head; tail -c $i tail ) | gzip -t &>/dev/null && { echo "TEST OK: $i"; break; }
done

() create subshell. Its output is fed to gzip -t and it would read from stdin, if it is not a terminal. No temp files - all data are in pipe.

In your case I think tail -c should be fine. Many GNU tools (tail included) have quite a lot of performance helping optimizations. E.g. on my SUSE in office, tail -c used mmap() to access the input file.

Otherwise for reading a file at an offset one normally uses dd.

P.S. In Perl you can read the head and tail files into memory and then using substr() try feeding pieces to some gzip library from CPAN. (I'm pretty sure there are gzip libraries for Perl - but I haven't used one. Google shows immediately few hits.) That way you would decrease overhead further by removing start of processes and rereading of files.

Answer 2

Here is a corrected version of your command:

cp head x.gz; tail -c $i tail >> x.gz; gzip -t x.gz >/dev/null 2>&1 

By redirecting all output to /dev/null then you're relying solely on the exit code of gzip for the result of the integrity test since no messages will be printed to the terminal. Note that the order of redirection is significant.

If you don't want to create an intermediate file:

cat head <(tail -c $i tail) | gzip -t >/dev/null 2>&1

It looks like that earlier in your script, you're creating files called "head" and "tail". It may be possible to do that differently and have a more efficient operation overall.