How to verify information using standard linux/unix filters?

Question

I have the following data in a Tab delimited file:

_ DATA _

Col1    Col2     Col3     Col4    Col5
blah1   blah2     blah3   4       someotherText
blahA   blahZ     blahJ   2       someotherText1
blahB   blahT     blahT   7       someotherText2
blahC   blahQ     blahL   10      someotherText3

I want to make sure that the data in 4th column of this file is always an integer. I know how to do this in perl

• Read each line, Store value of 4th column in a variable
• check if that variable is an integer
• if above is true, continue the loop
• else break out of the loop with message saying file data not correct

But how would I do this in a shell script using standard linux/unix filter? My guess would be to use grep, but I am not sure how?

Answer 1

awk is the tool most naturally suited for parsing by columns:

awk '{if ($4 !~ /^[0-9]+$/) { print "Error! Column 4 is not an integer:"; print $0; exit 1}}' data.txt

As you get more complex with your error detection, you'll probably want to put the awk script in a file and invoke it with awk -f verify.awk data.txt.

Edit: in the form you'd put into verify.awk:

{
    if ($4 !~/^[0-9]+$/)  {
        print "Error! Column 4 is not an integer:"
        print $0
        exit 1
    }
}

Note that I've made awk exit with a non-zero code, so that you can easily check it in your calling script with something like this in bash:

if awk -f verify.awk data.txt; then
     # action for success
else
     # action for failure
fi

You could use grep, but it doesn't inherently recognize columns. You'd be stuck writing patterns to match the columns.

Answer 2

cut -f 4 filename

will return the fourth field of each line to stdout.

Hopefully that's a good start, because it's been a long time since I had to do any major shell scripting.

Answer 3

Edited....

#!/bin/bash

isdigit ()
{
    [ $# -eq 1 ] || return 0

    case $1 in
        *[!0-9]*|"") return 0;;
        *) return 1;;
    esac
}

while read line
do
    col=($line)
    digit=${col[3]}

    if isdigit "$digit"
    then
        echo "err, no digit $digit"
    else
        echo "hey, we got a digit $digit"
    fi
done

Use this in a script foo.sh and run it like ./foo.sh < data.txt

See tldp.org for more info

Answer 4

cut -f4 data | LANG=C grep -q '[^0-9]' && echo invalid

• LANG=C for speed
• -q to quit at first error in possible long file

If you need to strip the first line then use tail -n+2 or you could get hacky and use:

cut -f4 data | LANG=C sed -n '1b;/[^0-9]/{s/.*/invalid/p;q}'

Answer 5

awk is what you need.

I can't upvote yet, but I would upvote Jefromi's answer if I could.

Answer 6

Mind, this may well not be the most efficient compared to iterating through the file with something like perl.

tail +2 x.x | sort -n -k 4 | head -1 | cut -f 4 | egrep "^[0-9]+$"
if [ "$?" == "0" ]
then
    echo "file is ok";
fi

tail +2 gives you all but the first line (since your sample has a header) sort -n -k 4 sorts the file numerically on the 4th column, letters will rise to the top. head -1 gives you the first line of the file cut -f 4 gives you the 4th column, of the first line egrep "^[0-9]+$" checks if the value is a number (integers in this case).

If egrep finds nothing, $? is 1, otherwise it's 0.

There's also:

if [ `tail +2 x.x | wc -l` == `tail +2 x.x | cut -f 4 | egrep "^[0-9]+$" | wc -l` ] then
    echo "file is ok";
fi

This will be faster, requiring two simple scans through the file, but it's not a single pipeline.

Answer 7

Sometimes you need it BASH only, because tr, cut & awk behave differently on Linux/Solaris/Aix/BSD/etc:

while read a b c d e ;  do [[ "$d" =~ ^[0-9] ]] || echo "$a: $d not a numer" ;  done < data

Answer 8

Pure Bash:

linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer"; fi; ((linenum++)); done < data.txt

To stop at the first error, add a break:

linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer" && break; fi; ((linenum++)); done < data.txt

Answer 9

@OP, use awk

awk '$4+0<=0{print "not ok";exit}' file