How would you group a number of lines?

Question

Let's say you have the following.

192.168.0.100
192.168.0.100
192.168.0.100
192.168.0.102
192.168.0.102
192.168.0.100

That's considered 3 unique hits. The way to distinguish it is that consecutive identical IPs count as one. How would you loop through the file and count accordingly?

Answer 1

Am not familiar with bash scripting, but the idea would be to keep track of the previous checked IP. Then if previous == current, don't increment, else increment?

Answer 2

I would avoid using bash for this. Use a real language like Python, awk or even Perl.

Python

#!/usr/bin/env python 
from __future__ import print_function
import fileinput
def combine( source ):
    count, prev= 1, source.next()
    for line in source:
        if line == prev:
            count += 1
        else:
            yield count, prev
            count, prev = 1, line
    yield count, prev
 for count, text in combine( fileinput.input() ):
    print( count, text )

Simple and extremely fast compared to bash.

Since this reads from stdin and writes to stdout, you can use it as a simple command in a pipeline.

Answer 3

If your uniq is like mine, and works only similar strings in sequence, just don't sort before your uniq:

file foo.txt:

192.168.0.100
192.168.0.100
192.168.0.100
192.168.0.102
192.168.0.102
192.168.0.100

And:

$ cat foo.txt | uniq -c

edit: can I give myself a useless use of cat award?

$ uniq -c foo.txt

/edit
Output:

  3 192.168.0.100
  2 192.168.0.102
  1 192.168.0.100

Answer 4

Similar to @Wrikken's answer, but I think you want total counts:

If your file containing the data above is foo.txt, then:

$ cat foo.txt | uniq | wc -l
3

Which is what you want I think.