views:

73

answers:

4

I am trying to find out the frequency of appearance of every letter in the english alphabet in an input file. How can I do this in a bash script?

+2  A: 

Here is a suggestion:

while read -n 1 c
do
    echo "$c"
done < "$INPUT_FILE" | grep '[[:alpha:]]' | sort | uniq -c | sort -nr
Benoit
Thank you for replying.
SkypeMeSM
+2  A: 

Just one awk command

awk -vFS="" '{for(i=1;i<=NF;i++)w[$i]++}END{for(i in w) print i,w[i]}' file

if you want case insensitive, add tolower()

awk -vFS="" '{for(i=1;i<=NF;i++)w[tolower($i)]++}END{for(i in w) print i,w[i]}' file

and if you want only characters,

awk -vFS="" '{for(i=1;i<=NF;i++){ if($i~/[a-zA-Z]/) { w[tolower($i)]++} } }END{for(i in w) print i,w[i]}' file

and if you want only digits, change /[a-zA-Z]/ to /[0-9]/

if you do not want to show unicode, do export LC_ALL=C

ghostdog74
Thanks you for your reply.
SkypeMeSM
I am sorry I am not very familiar with awk. The solution works but I am getting all characters instead of just alphanumeric characters. awk -vFS="" '{for(i=1;i<=NF;i++)w[tolower($i)]++ sum++ } END{for(i in w) print i,w[i],w[i]/sum}'
SkypeMeSM
Thanks again. I am wondering why I get results like ü 2 and é 2, when the regex is [a-zA-Z].
SkypeMeSM
that's because gawk's regex works for unicode characters. (UTF8).
ghostdog74
how can i remove them in that case?
SkypeMeSM
you can do a `export LC_ALL=C`.
ghostdog74
Yes worked like a charm. Thanks.
SkypeMeSM
+3  A: 

A solution with sed, sort and uniq:

sed 's/\(.\)/\1\n/g' file | sort | uniq -c

This counts all characters, not only letters. You can filter out with:

sed 's/\(.\)/\1\n/g' file | grep '[A-Za-z]' | sort | uniq -c

If you want to consider uppercase and lowercase as same, just add a translation:

sed 's/\(.\)/\1\n/g' file | tr '[:upper:]' '[:lower:]' | grep '[a-z]' | sort | uniq -c
mouviciel
Thanks. This considers uppercase and lowercase characters as separate. How can I calculate the frequencies where we consider A and a as same?
SkypeMeSM
Yes this works great as well. I am wondering how can I calculate the probabilities i.e. frequency/total sum. We will need to pipe the output again to sed again but I cannot figure out the regex involved?
SkypeMeSM
You can add some `wc`, `cut`, `dc`, `tee` and other commands but it would be more juggling with plates than a maintainable work. I think that adding more features would be easier with a perl script.
mouviciel
Thank you very very much for your help. Cheers.
SkypeMeSM
A: 

My solution using grep, sort and uniq.

grep -o . file | sort | uniq -c

Ignore case:

grep -o . file | sort -f | uniq -ic
dogbane
how can I get frequency / sum(all frequencies) after this?
SkypeMeSM