ansaurus

Question

creating tidy shell script from ugly command line pipeline

Answer 1

+3 A:

Pipes don't stop you when breaking to multiple lines, but use $( ... ) instead of backticks. Something like this should work:

#!/bin/bash

for number in $(
    cat xmlEventLog_2010-03-23T* |
    sed -nr "/<event eventTimestamp/,/<\/event>/ {/event /{s/^.*$/\n/; p};/payloadType / {h; /protocol/ {s/.*protocol=\"([^\"]*)?\".*/protocol: \1/}; p; x; /type/ {s/.*type=\"([^\"]+)\".*/payload: \1/g}; /type/! {s/.*protocol=\"([^\"]+)\".*/payload: \1/g}; p};/sender / {/sccpAddress/ {s/.*sccpAddress=\"([^\"]*)?\".*/sccpAddress: \1/}; /sccpAddress/! {s/.*/sccpAddress: Unknown/}; p};/result /{s/.*value=\"([^\"]+)\".*/result: \1/g; p};/filter code/{s/.*type=\"([^\"]+)\".*/type: \1/g; p};}"|
    tee checkThis.txt |
    awk 'BEGIN{FS="\n"; RS=""; OFS=";"; ORS="\n"} $1~/result: Blocked|Modified/ && $2~/sccpAddress: 353201000001/ && $4~/payload: SMS-MO-FSM-INFO|SMS-MO-FSM/ {$1=$1 ""; print}' |
    sort |
    uniq -c |
    egrep "NUMBER_BLACKLIST|USER_BLACKLIST|NUMBER_WALLEDGARDEN|USER_WALLED_GARDEN|SERVICE_RESTRICTION|BLOCK_VOICE_TO_SMS|PEP_Blacklist_Whitelist" |
    awk '{print $1}'
  ); do fil="$fil+$number"
done
echo "fil is $fil"

Of course the larger part is to split the awk and sed skripts into multiple lines also...

But I believe that even after that the result will be still quite unreadable.

I would suggest just completely rewriting the script in Perl, Ruby or any other a bit more readable scripting language than Bash. This is just a suggestion from my personal experience - every time a start out with a shell script I finally rewrite it in Ruby. I love Bash, but it just doesn't seem to scale.

Rene Saarsoo 2010-09-23 13:36:05

I wrote it in python too (very pretty) but I like to have the bash version cleaned up. Thanks

amadain 2010-09-23 13:57:29

What's wrong with backticks?

Roman Cheplyaka 2010-09-23 17:28:05

I'm with Rene. You need to use something that is geared for working with XML files. You shouldn't try to use regexes for that. Short of that, the `sed` part could be rewritten in AWK, the `egrep` and the `uniq` could be done in AWK. If you have `gawk`, you could do the `sort`, too. But in the end, you should use a Python or Perl XML module.

Dennis Williamson 2010-09-23 17:46:27

@Roman: Backticks: [BashFAQ/082](http://mywiki.wooledge.org/BashFAQ/082)

Dennis Williamson 2010-09-23 17:48:55

@Dennis: I agree about "preferred", but the original answer gives an impression that backticks won't work with wrapped lines, which is false.

Roman Cheplyaka 2010-09-23 18:21:21

How would you write that script entirely in awk? I wrote this with python and xml.sax but for various reasons I needed to have a shell version of it.

amadain 2010-09-26 18:16:46

Answer 2

+2 A:

Two small remarks:

Put the 'for list' in a separate function:

number_list() {
    # complete pipe command list
    # divided over multiple lines
}

for number in `number_list`
do
   # ...
done

Try to combine some of the commands: The cat is not needed, the final egrep and awk can be combined.

schot 2010-09-23 13:39:46

why is the cat not needed?

amadain 2010-09-23 14:20:21

@amadain: `sed` also takes multiple file arguments, so you can replace `cat file ... | sed [pat]` with `sed [pat] file ...`.

schot 2010-09-23 14:30:16

Answer 3

+1 A:

You can join the different tokens using tr and prepend 'fil is' using sed:

pipeline | tr -s ' \n\t' '+' | sed -e '1s/^/fil is /' -e '$s/+$//'

The pipeline can be split over multiple lines using \:

first-command \
    | second-command \
    | third-command \
    ...
    | last-command

Bart Sas 2010-09-23 13:59:48

Answer 4

+1 A:

The shell script is actually the simple part. The sed script is the scary bit. The script can be improved with here documents, but witness the comment:

#!/bin/bash

seds=/tmp/seds.$$
awks=/tmp/awks.$$
gres=/tmp/gres.$$

trap "rm -f $seds $awks $gres" 0 1 2 3 15

# this is a noble and hairy attempt to parse xml with sed
# it is extremely fragile and strongly dependent upon
# the form of the source file never changing
# I'm alternately proud or disgusted that I've been able
# to get away with this

cat > $seds <<'EOF'
/<event eventTimestamp/,/<\/event>/ {/event /{s/^.*$/\n/; p};
/payloadType / {h; /protocol/ {s/.*protocol=\"([^\"]*)?\".*/protocol: \1/}; p; x;
/type/ {s/.*type=\"([^\"]+)\".*/payload: \1/g};
/type/! {s/.*protocol=\"([^\"]+)\".*/payload: \1/g}; p};
/sender / {/sccpAddress/ {s/.*sccpAddress=\"([^\"]*)?\".*/sccpAddress: \1/};
/sccpAddress/! {s/.*/sccpAddress: Unknown/}; p};
/result /{s/.*value=\"([^\"]+)\".*/result: \1/g; p};
/filter code/{s/.*type=\"([^\"]+)\".*/type: \1/g; p};}
EOF

cat > $awks <<'EOF'
BEGIN {FS="\n"; RS=""; OFS=";"; ORS="\n"}
$1~/result: Blocked|Modified/ && \
$2~/sccpAddress: 353201000001/ && \
$4~/payload: SMS-MO-FSM-INFO|SMS-MO-FSM/ {$1=$1 ""; print}
EOF

cat > $gres <<EOF
NUMBER_BLACKLIST
USER_BLACKLIST
NUMBER_WALLEDGARDEN
USER_WALLED_GARDEN
SERVICE_RESTRICTION
BLOCK_VOICE_TO_SMS
PEP_Blacklist_Whitelist
EOF

cat xmlEventLog_2010-03-23T* | \
sed -nr -f $seds | \
tee checkThis.txt | \
awk -f $awks | \
sort | uniq -c | \
fgrep -f $gres | \
awk '{print $1}'

msw 2010-09-23 14:55:29

ansaurus

tags:

views:

answers:

creating tidy shell script from ugly command line pipeline

related questions