Best Awk Commands

Question

I find AWK really useful. Here is a one liner I put together to manipulate data.

ls | awk '{ print "awk " "'"'"'"  " {print $1,$2,$3} " "'"'"'"  " " $1 ".old_ext > " $1    ".new_ext"  }' > file.csh

I used this AWK to make a script file that would rename some files and only print out selective columns. Anyone know a better way to do this? What are you best AWK one liners or clever manipulations?

Answer 1

Henry Spencer wrote a fairly good implementation of nroff on awk. He called it "awf". He also claimed that if Larry Wall had known how powerful awk was, he wouldn't have needed to invent perl.

Answer 2

Here's a couple of awks that I used to use regularly ... note that you can use $1, $2, etc to get out the column you want. So, for manipulating a bunch of files, for example here's a stupid command you could use instead of mv ...

ls -1 *.mp3 | awk '{printf("mv %s newDir/%s\n",$1,$1)}' | /bin/sh

Or if you're looking at a set of processes maybe ...

ps -ef | grep -v username | awk '{printf("kill -9 %s\n",$2)}' | /bin/sh

Pretty trivial but you can see how that would get you quite a ways. =) Most of the stuff I used to do you can use xargs for, but hey, who needs them new fangled commands?

Answer 3

The AWK book is full of great examples. You can download them all. (The first couple lines of Awk code at that link are an unarchiver for the remainder.)

Answer 4

I use this script a lot for editing PATH and path-like environment variables. Usage:

export PATH=$(clnpath /new/bin:/other/bin:$PATH /old/bin:/other/old/bin)

This command adds /new/bin and /other/bin in front of PATH, removes both /old/bin and /other/old/bin from PATH (if present - no error if absent), and removes duplicate directory entries on path.

:   "@(#)$Id: clnpath.sh,v 1.6 1999/06/08 23:34:07 jleffler Exp $"
#
#   Print minimal version of $PATH, possibly removing some items

case $# in
0)  chop=""; path=${PATH:?};;
1)  chop=""; path=$1;;
2)  chop=$2; path=$1;;
*)  echo "Usage: `basename $0 .sh` [$PATH [remove:list]]" >&2
    exit 1;;
esac

# Beware of the quotes in the assignment to chop!
echo "$path" |
${AWK:-awk} -F: '#
BEGIN       {       # Sort out which path components to omit
                    chop="'"$chop"'";
                    if (chop != "") nr = split(chop, remove); else nr = 0;
                    for (i = 1; i <= nr; i++)
                            omit[remove[i]] = 1;
            }
{
    for (i = 1; i <= NF; i++)
    {
            x=$i;
            if (x == "") x = ".";
            if (omit[x] == 0 && path[x]++ == 0)
            {
                    output = output pad x;
                    pad = ":";
            }
    }
    print output;
}'

Answer 5

I use this:

df -m | awk '{p+=$3}; END {print p}'

To total all disk space used on a system across filesystems.

Answer 6

You can find several nice one liners here.

Answer 7

Many years ago I wrote a tail script in awk:

#!/usr/bin/awk -f
BEGIN {
  lines=10
}

{
  high = NR % lines + 1
  a[high] = $0
}

END {
  for (i = 0; i < lines; i++) {
    n = (i + high) % lines + 1
    if (n in a) {
      print a[n]
    }
  }
}

It's silly, I know, but that's what awk does to you. It's just very fun playing with it.

Answer 8

Count memory used by httpd

ps -ylC httpd | awk '/[0-9]/ {SUM += $8} END {print SUM/1024}'

Or any other process by replacing httpd. Dividing by 1024 to get output in MB.