awk

Problem removing duplicate files using awk

Contents of part3.1.awk { current_line=$0 if (current_line!=prev) { print $1 " -> " " -> " $5 " -> " $8 } prev=$0 } To get the list of processes, i run this in terminal. I want to get output with removed duplicates and sorted too. $ps -ef | awk -f part3.1.awk | sort What wrong am i doing? ...

Java exec() does not return expected result of pipes' connected commands

I'm calling command line programs connected by pipes. All this works on Linux for sure. My method: protected String execCommand(String command) throws IOException { String line = null; if (command.length() > 0) { Process child = Runtime.getRuntime().exec(command); InputStream lsOut = child.getInputStream(); ...

Compare file with variable list AWK

I'm stumbling over myself trying to get a seemingly simple thing accomplished. I have one file, and one newline delimited string list. File: Dat1 Loc1 Dat2 Loc1 Dat3 Loc1 Dat4 Loc2 Dat5 Loc2 My list is something like this: Dat1 Dat2 Dat3 Dat4 What I am trying to do is compare the ...

How to use awk to aggregate rows of data

Hi, I have a question, I have a set of data in rows which some rows are belong to a group. E.g Apple 0.4 0.5 0.6 Orange 0.2 0.3 0.2 Apple 0.4 0.3 0.4 Orange 0.4 0.5 0.8 The question is how can I automatically aggregate the columns accordingly using awk. In the past, I would easily deal with the following awk manually for each file....

print second last column/field in AWK

I want to print the second last column or field in awk. the number of fields is variable. I know that I should be able to use $NF but not sure how it can be used. And this does not seem to work awk ' { print ( $NF-- ) } ' ...

Linux Text File Manipulation

Hi, I have a file of the format: <a href="http://www.wowhead.com/?search=Superior Mana Oil"> <a href="http://www.wowhead.com/?search=Tabard of Brute Force"> <a href="http://www.wowhead.com/?search=Tabard of the Wyrmrest Accord"> <a href="http://www.wowhead.com/?search=Tattered Hexcloth Sack"> I need to select the text after the = but ...

Linux Text File Manipulation with sed/awk

Hi, I have a list in the following format 77 Infinite Dust 4 Illusion Dust 12 Dream Shard 29 Star's Sorrow I need to change this to: 77 <a href="http://www.wowhead.com/?search=Infinite Dust">Infinite Dust</a> 4 <a href="http://www.wowhead.com/?search=Illusion Dust">Illusion Dust</a> 12 <a href="http://www.wowhead.com/?search=Dream...

Combine Text Lines That Match Criteria, Linux/sed/awk?

Hi I have a file like the following: 1, cake:01351 12, bun:1063 scone:13581 biscuit:1931 14, jelly:1385 I need to convert it so that when a number is read at the start of a line it is combined with the line beneath it, but if there is no number at the start the line is left as is. This would be the output that I need: 1,cake:01351 12...

replace a particular value in a string of key - values

I have a large string which is a collection of key (space)+ value, i.e key followed by one or more space and then value. Now i need to change the value of a particular key using sed, awk, grep etc. in unix environment. eg. of string: -Key1 Value1 -Key2 Value2 -Key3 Value3 I need a new string that is same as above only Va...

unix shell script to remove blank lines following opening brackets

Hi all, I need a (sed, awk) shell script or, even better, a vim command to remove any blank lines following a line with a single opening curly bracket: void func() { foo(); } void bar() { helloWorld(); } should become: void func() { foo(); } void bar() { helloWorld(); } Any thoughts? ...

Processing apache logs quickly

Hi! I'm currently running an awk script to process a large (8.1GB) access-log file, and it's taking forever to finish. In 20 minutes, it wrote 14MB of the (1000 +- 500)MB I expect it to write, and I wonder if I can process it much faster somehow. Here is the awk script: #!/bin/bash awk '{t=$4" "$5; gsub("[\[\]\/]"," ",t); sub(":"," ...

How to debug an awk script

Hi, may i know the tips for debugging any awk script? Please feel free to share if you have any. Thanks ...

Converting dates in AWK

I have a file containing many columns of text, including a timestamp along the lines of "Fri Jan 02 18:23" and I need to convert that date into MM/DD/YYYY HH:MM format. I have been trying to use the standard 'date' tool with awk getline to do the conversion, but I can't quite figure out how to pass the fields into the 'date' command in ...

How to output awk result to file

Hi I am trying to output 'awk' result to file in my script, with no success. Using '>' does not work, why? for a in $(find $OUPUT_DIR/ -maxdepth 1 -mindepth 1 -type d -printf "%P\n") do echo $a is a directory awk -F, '{ if ($10 == '"$a"') print $0 }' $OUPUT_DIR/CDRNOutput_${CDR_DATE}.csv > $OUPUT_DIR/$a/CDR-${CDR_DATE}.csv done...

Remove all lines between two strings

In a sh shell script. Given data in a text file: string1 string2 gibberish gibberish string3 gibberish string4 How could you use awk or sed to remove all lines between string2(inclusive) and string3(not including string 3)? to end up with: string1 string3 string4 ...

Bash or Python or Awk to match and modify files

Hi, I have a set of 10000 files c1.dat ... c10000.dat. Each of these files contains a line which starts with @ and contains a string with spaces specific for this file, lije c37 7.379 6.23. I have another set of 10000 files kind of determined_cXXX_send.dat (where XXX goes from 1 to 10000). Each of these files has only one line. Each l...

AWK command help

How do I write an awk command that reads through the /etc/passwd file, and prints out just the names of any users who have the /bin/bash program as their default command shell? ...

Control Processes Started by Bash Daemon

In bash, I have created a simple daemon to execute commands when my internet connection changes: #!/bin/bash doService(){ while do checkTheInternetConnection sleep 15 done } checkTheInternetConnection(){ if unchanged since last check return else execute someCommand fi } someCommand(){...

Modifying Text Column Wise with Sed/AWK

I have an input data with three columns (tab separated) like this: a mrna_185598_SGL 463 b mrna_9210_DLT 463 c mrna_9210_IND 463 d mrna_9210_INS 463 e mrna_9210_SGL 463 How can I use sed/awk to modify it into four columns data that looks like this: a mrna_185598 SGL 463 b mrna_9210 DLT 463 c mrna_9210 ...

How can I shuffle the lines of a text file in Unix command line?

I searched SO for a similar Q/A to no avail. I want to shuffle the lines of a text file randomly and create a new file. The file may have several thousands of lines. How can I do that with cat, awk, cut, etc.? ...