Get duplicate ids with bash

Question

I have a log file with format

id operation

id success message

The second id is same as the first, but if only the the operation is successful. How can I get these two lines from the log to a new file?

Answer 1

it's primative but you could do something like this quite easily:

cat file.log | cut -d " " -f1 | uniq -d > succesfulOperationIDs.dat

while read line
do
     grep ^$line file.log > successLog.log
done < succesfulOperationIDs.dat

rm -f succesfulOperationIDs.dat

This creates a list of id's that appear more than once (assuming a space is the right delimeter) then puts all lines beginning with those id's into a new file.

Answer 2

Give this a try:

awk '{if ($1 == previd) {print prevline; print} else {previd = $1; prevline = $0}}' logfile > newfile

It assumes that success messages immediately follow operation messages.

If that's not the case, then try this:

awk '{if (ids[$1]) {print lines[$1]; print} else {ids[$1] = $1; lines[$1] = $0}}' logfile > newfile

If operations and success messages appear in pairs more than once with the same ID, you might want to reset the tracking each time the pair is output:

awk '{if (ids[$1]) {print lines[$1]; print; delete ids[$1]; delete lines[$1]} else {ids[$1] = $1; lines[$1] = $0}}' logfile > newfile