problems combining files using unix cat and zcat in shell script

Question

I've got a problem with my shell script. I am basically trying to create a new log file from (2) files using unix (zcat, cat) commands

FileA = gzipped logfile (.gz)
FileB = non-gzipped log file (.log)

FileC = new file from FileA:FileB

My problem is with FileC.

For example: FileA has timestamp data from Aug19-Sept3 FileB has timestamp data from Sept4-Sept17 FileC has contents from both files = Aug19-Sept3:Sept4-Sept17

Problem: If I manually run the commands on the server, the file looks fine as I expect FileC to have timestamp data from Aug19-Sept17 But, if done programmatically with the shell script, the local copy looks like this when I copy it back remotely: FileC = Aug19-Sept17:Aug19-Sept2

So my question is why is this happening programmatically? Is there a better way to combine/stitch these 2 logs together to create another one?

Here is my shell script:

#!/bin/bash
if [ $# != 1 ]; then
    echo "Usage: getlogs.sh <remote-host>" 2>&1
    exit 1
fi

#Declare variables
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"

#Copy file over
echo "START..." 2>&1
test -f $BACKUP_DIR$1.mysoft.log
if [ $? = 0 ]; then
   echo "Local log file $BACKUP_DIR$1.mysoft.log exists, clean up for new copy..." 2>&1
   /bin/rm $BACKUP_DIR$1.mysoft.log
   else
        echo "File does not exist, getting a new copy..." 2>&1
fi

echo "Checking remotely in $1 for logfile $REMOTE_MYCNF $STAMP" 2>&1
if [ ! -f $REMOTE_MYCNF ]; then
   echo "File exists remotely, creating new logfile and copy here...." 2>&1
   ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
   ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"
   /usr/bin/scp $1:$REMOTE_DIR$NEWLOG $BACKUP_DIR$1.mysoft.log
   echo "end remote copy" 2>&1
   echo "Cleaning up remote files" 2>&1
   ssh $1 "rm $REMOTE_DIR$NEWLOG"
   exit 0
   else
        echo "Unable to get file" 2>&1
        exit 0
fi

Answer 1

I can't guess immediately what the error is, but have you tried simplifying your shell script? That is, start with this:

#!/bin/sh
STAMP=`date '+%Y%m%d-%H:%M'`
REMOTE_MYCNF=/var/log/mysoft/mysoft.log
REMOTE_GZ=/var/log/mysoft/mysoft.log.1.gz
REMOTE_DIR=/var/log/mysoft/
BACKUP_DIR=/home/dev/logs/
NEWLOG="foo-temp.log"
ssh $1 "zcat $REMOTE_GZ >> $REMOTE_DIR$NEWLOG"
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"

Then build up from here. This will at least allow you to isolate the problem.

Answer 2

Perhaps it's possible that the cat is starting before the zcat is finished. However, it would surprise me if this were the case.

Try this:

ssh $1 "zcat ... >> ...; cat ... >> ..."

or

ssh $1 "zcat ... >> ..."
sleep 5
ssh $1 "cat ... >> ..."

or

ssh $1 "zcat -f file1 file2 >> ..."   # let zcat do both files

Answer 3

It seems the temp file is getting reused, delete or truncate it:

# Always using the same log for new log.
NEWLOG="foo-temp.log"
# Use a new log truncate '>' the old one
ssh $1 "zcat $REMOTE_GZ > $REMOTE_DIR$NEWLOG"
# This time, append '>>' to new log
ssh $1 "cat $REMOTE_MYCNF >> $REMOTE_DIR$NEWLOG"

Answer 4

You don't need to create a remote temporary file at all. Get rid of the multiple ssh commands, scp, and cleanup ssh, and try this instead:

ssh $1 "zcat $REMOTE_GZ; cat $REMOTE_MYCNF;" > $BACKUP_DIR$1.mysoft.log

zcat and cat will write data to their stdout, which is attached to ssh's stdout, which we redirect to a file.

I "guarantee" that zcat writes all its data and quits before cat begins, giving the desired ordering. "guarantee" is in quotes because the other solutions should work, which are reported not to.