awk

Replace delimited block of text in file with the contents of another file

I need to write a simple script to replace a block of text in a configuration file with the contents of another file. Let's assume with have the following simplified files: server.xml <?xml version='1.0' encoding='UTF-8'?> <Server port="8005" shutdown="SHUTDOWN"> <Service name="Catalina"> <Connector port="80" protocol="HTTP/1.1"...

How can I quickly sum all numbers in a file?

I have a file which contains several thousand numbers, each on it's own line: 34 42 11 6 2 99 ... I'm looking to write a script which will print the sum of all numbers in the file. I've got a solution, but it's not very efficient. (It takes several minutes to run.) I'm looking for a more efficient solution. Any suggestions? ...

I cant really wrap my head around BOOLEAN logic when I use NOT together with AND and OR

Im trying to understand how boolean logic works when I use NOT. To give an example using awk I have a text file containing CORE PORT CORE PORT COREPORT CORE COREPORT And I would like to remove all COREPORT lines. The way I thought I would do it was with (NOT CORE) AND (NOT PORT) eg awk '/!CORE/&&/!PORT/{print}' But when I try it...

How to use regex to match ASTERISK in awk

I'm stil pretty new to regular expression and just started learning to use awk. What I am trying to accomplish is writing a ksh script to read-in lines from text, and and for every lines that match the following: *RECORD 0000001 [some_serial_#] to replace $2 (i.e. 000001) with a different number. So essentially the script read ...

awk/sed/bash to merge/concatenate data

Trying to merge some data that I have. The input would look like so: foo bar foo baz boo abc def abc ghi And I would like the output to look like: foo bar baz boo abc def ghi I have some ideas using some arrays in a shell script, but I was looking for a more elegant or quicker solution. ...

substitute string in php from another file

i have some old php files which i'd like to convert to use gettext. those files have a content like this : $LD = 'Some String'; $Another = 'some other ~n~ string'; i have to substitute all the $LD, $Another in the files where they are declared with something like : _('Some string'); hacking a bit a created some sort of regexp to f...

search for a string , and add if it matches

I have a file that has 2 columns as given below.... 101 6 102 23 103 45 109 36 101 42 108 21 102 24 109 67 and so on...... I want to write a script that adds the values from 2nd column if their corresponding first column matches for example add all 2nd column values if it's 1st column is 101 add all 2nd column values...

Processing a tab delimited file with shell script processing

Hello, normally I would use Python/Perl for this procedure but I find myself (for political reasons) having to pull this off using a bash shell. I have a large tab delimited file that contains six columns and the second column is integers. I need to shell script a solution that would verify that the file indeed is six columns and that t...

Counting unique values in a column with a shell script

Hello. I have a tab delimited file with 5 columns and need to retrieve a count of just the number of unique lines from column 2. I would normally do this with Perl/Python but I am forced to use the shell for this one. I have successfully in the past used *nix uniq function piped to wc but it looks like I am going to have to use awk in h...

bash split text into limited character buckets (array member)

i have text such as http://pastebin.com/H8zTbG54 we can say this text is set of rules splitted by "OR" at the end of lines i need to put set of lines(rules) into buckets (bash array members) but i have character limit for each array member which is 1024 so each array member should contain set of rules but character count for each ...

join 3 files by first Column with join (was awk) ?

i have three similar files, they are all like this: File A ID1 Value1a ID2 Value2a . . . IDN Value2n and i want an output like this Output ID1 Value1a Value1b Value1c ID2 Value2a Value2b Value2c ..... IDN ValueNa ValueNb ValueNc Looking to the first line, i want value1A to be the value of id1 in fileA, value1B the value of ...

Optimizing grep (or using AWK) in a shell script

Hi - In my shell script, I am trying to search using terms found in a $sourcefile against the same $targetfile over and over. My $sourcefile is formatted as such: pattern1 pattern2 etc... The inefficient loop I have to search with is: for line in $(< $sourcefile);do fgrep $line $targetfile | fgrep "RID" >> $outputfile done I ...

AWK: compare apache dates without using regular expression

I'm writing a loganalysis application and wanted to grab apache log records between two certain dates. Assume that a date is formated as such: 22/Dec/2009:00:19 (day/month/year:hour:minute) Currently, I'm using a regular expression to replace the month name with its numeric value, remove the separators, so the above date is converted to...

awk / sed script to remove text

Hi, I am currently in need of a way to programmatically remove some text from Makefiles that I am dealing with. Now the problem is that (for whatever reason) the makefiles are being generated with link commands of -l<full_path_to_library>/<library_name> when they should be generated with -l<library_name>. So what I need is a script to f...

php script dies when it calls a bash script, maybe a problem of server configuration

Hi!!! I have some problems with a PHP script that calls a Bash script.... in the PHP script is uploaded a XML file, then the PHP script calls a Bash script that cut the file in portions (for example, is uploaded a XML file of 30,000 lines, so the Bash script cut the file in portions of 10,000 lines, so it will be 3 files of 10,000 each ...

Extract Lines when Column K is empty with AWK/Perl

I have data that looks like this: foo 78 xxx bar yyy qux 99 zzz xuq xyz They are tab delimited. How can I extract lines where column 2 is empty, yielding bar yyy xuq xyz I tried this but doesn't seem to work: awk '$2==""' myfile.txt ...

How to number the ls output in unix?

I am trying to write a file with format - "id file_absolute_path" which basically lists down all the files recursively in a folder and give an identifier to each file listed like 1,2,3,4. I can get the absolute path of the files recursively using the following command: ls -d -1 $PWD/**/*/* However, I am unable to give an identifier ...

shell script filter du and find by a string inside a file in a subfolder

I have the following command that I run on cygwin: find /cygdrive/d/tmp/* -maxdepth 0 -mtime -150 -type d | xargs du --max-depth=0 > foldersizesreport.csv I intended to do the following with this command: for each folder under /d/tmp/ that was modified in last 150 days, check its total size including files within it and ...

How do I get Git's latest stable release version number?

I'm writing a git-install.sh script: http://gist.github.com/419201 To get Git's latest stable release version number, I do: LSR_NUM=$(curl -silent http://git-scm.com/ | sed -n '/id="ver"/ s/.*v\([0-9].*\)<.*/\1/p') 2 Questions: Refactor my code: Is there a better way programmatically to do this? This works now, but it's brittle: if...

sed or grep or awk to match very very long lines

more file param1=" 1,deerfntjefnerjfntrjgntrjnvgrvgrtbvggfrjbntr*rfr4fv*frfftrjgtrignmtignmtyightygjn 2,3,4,5,6,7,8, rfcmckmfdkckemdio8u548384omxc,mor0ckofcmineucfhcbdjcnedjcnywedpeodl40fcrcmkedmrikmckffmcrffmrfrifmtrifmrifvysdfn drfr4fdr4fmedmifmitfmifrtfrfrfrfnurfnurnfrunfrufnrufnrufnrufnruf"**** need to match the content of p...