awk

How do I use awk to parse a fixed-width (NACHA) file format?

My company has a problem: we suspect that the NACHA files we are receiving from one of our application service providers that we use to draw money from our clients are incorrect. We have all of the ACH agreements and legal mumbo-jumbo in place, so it's not a problem with our use of the ACH network, and we're not receiving word from the...

sed and awk one liners

FYI below website is a great place for sed & awk one liners www.readylines.com Hope that helps. ...

How to remove lowercase sentence fragments from text?

Hello: I'm tyring to remove lowercase sentence fragments from standard text files using regular expresions or a simple Perl oneliner. These are commonly referred to as speech or attribution tags, for example - he said, she said, etc. This example shows before and after using manual deletion: Original: "Ah, that's perfectly tr...

Remove line from a file using a variable line number

Hi Guys, This is probably a simple one to answer, but I'm stuck, so here goes. sed '3d' filename # (or something like that) I'm having trouble trying to use a $VARIABLE instead of the number. Anyone know how to get this to work, or any alternative options? Regards Paul ...

Howto Pass A String as Parameter in AWK within Bash Script

I have a text file which I want to filter using awk. The text file looks like this: foo 1 bar 2 bar 0.3 bar 100 qux 1033 I want to filter those files with awk inside a bash script. #!/bin/bash #input file input=myfile.txt # I need to pass this as parameter # cos later I want to make it more general like # coltype=$1 col1type="foo" ...

sort associative array in awk - help?

Hi all, I have an associative array in awk that gets populated like this... chr_count[$3]++ When I try to print my chr_counts I use this: for (i in chr_count) { print i,":",chr_count[i]; } But not surprisingly, the order of i is not sorted in any way. Is there an easy way to iterate over the sorted "keys" of chr_count? ...

awk output to a file

I need help in moving the contents printed by awk to a text file. THis is a continuation of previous quesion I have to move all the contents into the same file so it is appending. To be specific nawk -v file="$FILE" 'BEGIN{RS=";"} /select/{ gsub(/.*select/,"select");gsub(/\n+/,"");print file,$0;} /update/{ gsub(/.*update/,"update")...

Combining multiple lines into one line

I have this use case of an xml file with input like Input: <abc a="1"> <val>0.25</val> </abc> <abc a="2"> <val>0.25</val> </abc> <abc a="3"> <val>0.35</val> </abc> ... Output: <abc a="1"><val>0.25</val></abc> <abc a="2"><val>0.25</val></abc> <abc a="3"><val>0.35</val></abc> I have around 200K lines in a file in the In...

parsing issue with comma separated csv file

I am trying to extract 4th column from csv file (comma separated, and skipping first 2 header lines) using this command, awk 'NR <2 {next}{FS =","}{print $4}' filename.csv | more However, it doesn't work because the first column cantains comma, thus 4th column is not really 4th. Below is an example of a row: "sdfsdfsd, sfsdf", 454,...

Messy bash variable

I'm writing a script to ssh in to a list of machines and compare a variable to another value.. I've run into a problem (I have a couple workarounds, but at this point I'm just wondering why this method isn't working). VAR=`ssh $i "awk -F: '/^bar/ {print \$2}' /local/foo.txt"` ($i would be a hostname. The hosts are trusted, no password...

How do I extract the values from this data using bash and awk?

I grepped these, how do I extract the values? ... cavity_2mgl_wt_strip57001.out: Total cavity volume (A3) : ( 1.240E+01) cavity_2mgl_wt_strip58001.out: Total cavity volume (A3) : ( 2.408E+00) cavity_2mgl_wt_strip60001.out: Total cavity volume (A3) : ( 4.935E+00) cavity_2mgl_wt_strip61001....

selecting the text string

ABCDchinchwad18-Mar-2010-11.sql.zip ABCDsolapur18-Mar-2010-10.sql.zip How do I find the string between "ABCD" and the date "18-Mar-2010" Expected resuts: chinchwad solapur ...

How to remove first comma on each line in a file

Remove first comma on each line in a file. I presume sed is needed. ...

Piping a bash variable into awk and storing the output

Hello, To illustrate my problem, TEST="Hi my name is John" OUTP=`echo $TEST | awk '{print $3}'` echo $OUTP What I would expect this to do is pass the $TEST variable into awk and store the 3rd word into $OUTP. Instead I get "Hi: not found", as if it is expecting the input to be a file. If I pass just a string instead of a variable,...

Use regex in awk command in bash script

I'm trying to read a file of regexes, looping over them and filtering them out of another file. I'm so close, but I'm having issues with my $regex var substitution I believe. while read regex do awk -vRS= '!/$regex/' ORS="\n\n" $tempOne > $tempTwo mv $tempTwo $tempOne done < $filterFile $tempOne and $tempTwo are temporary files. ...

grep for value of keyvaue pair and format

When I do the following ps -aef|grep "asdf" I get a list of processes that are running. Each one of my process has the following text in the output: -ProcessName=XXXX I'd like to be able to format the out put so all I get is: The following processes are running: Process A Process B etc.. ...

How to print ASCII value of a character using basic awk only.

I need to print the ASCII value of the given charecter in awk only. the below gives 0 as output. echo a | awk '{ printf("%d \n",$1); }' Help please. ...

READING stderr from within Awk

I want to keep SSH debug info separate (and logged) from other input. However, if I simply redirect stderr to a log file, I risk combining output from SSH and output from the remote process on the host machine (that might send something to stderr): $ ssh -v somemachine 2> file.log So, I want to filter out only those lines that match "...

Using sed and/or awk, how can do a sed swap on all text in a file until I see a line matching a regex?

How can I do a sed regex swap on all text that preceed a line matching a regex. e.g. How can I do a swap like this s/foo/bar/g for all text that precedes the first point this regex matches: m/baz/ I don't want to use positive/negative look ahead/behind in my sed regex, because those are really expensive operations on big files. ...

What is Perl's equivalent to awk's /text/,/END/ ?

I am looking to replace a nasty shell script that uses awk to trim down some HTML. The problem is I cannot find anything in Perl that does the aforementioned function awk '/<TABLE\ WIDTH\=\"100\%\" BORDER\=1\ CELLSPACING\=0><TR\ class\=\"tabhead\"><TH>State<\/TH>/,/END/' How can I do this in Perl? the expected output would be <TABLE...