ansaurus

Question

sed script to remove file name duplicates

Answer 1

A:

Sed probably isn't the best tool here as it's stream-oriented. You could possibly use it to convert the spaces to newlines though, pipe that through sort and uniq, then use sed again to convert the newlines back to spaces.

Typing this on my phone, though, so can't give exact commands :(

pdbartlett 2010-05-19 16:50:32

Looking again at the requirements you might need grep in there as well to just select the headers, and use one of the sed commands to prepend the src prefix. By now I'd be reaching for Perl, Python, Ruby, etc. though...

pdbartlett 2010-05-19 16:56:03

Answer 2

+2 A:

Not sed but hope this helps you:

make libs-depends | grep -io --perl-regexp "[\w\.\/]+\.h " | sort -u | sed -e 's:^:src/:'

Anton 2010-05-19 16:59:21

I would replace "sort | uniq" by "sort -u"

Nicolas Viennot 2010-05-19 17:10:58

@Pafy: Old habits die hard :) Replaced it.

Anton 2010-05-19 17:13:11

@Anton: Nice! Minor thing: how to join all lines? I believe that `sed -e 's/\n/ /g'` will not work as I need to accumulate all rows into buffer before running regex.

dma_k 2010-05-20 08:10:22

@dma_k, Maybe not relevant anymore but this a quick and easy way to join lines `tr '\n' ' '`. This will replace the new line character \n with space.

Anton 2010-05-20 08:23:57

@Anton: I found a small bug in a script: regexp should be with a leading space `' [\w\.\/]+\.h'`, because in the input there might be lines like `detect.o: detect.h\n`, so the leading space is guaranteed, but not the trailing. Unfortunately, I cannot adjust sed replacement: `sed -e 's:^ :src/:'` does not work. Please help.

dma_k 2010-05-20 09:51:43

@dma_k: As initial whitespace is optional you can use:`sed -e 's:^\s*:src/:'`

Anton 2010-05-20 10:07:52

@Anton: I found the root of the problem: I have `grep` aliased to `grep --color=always` which always highlights the match found. This causes the input for `sed` to be like this `00000000 1b 5b 30 31 3b 33 31 6d 1b 5b 4b 20 61 6d 69 67 |.[01;31m.[K amig|` and of course, sed cannot match the beginning of the line. From script it works as a charm.

dma_k 2010-05-21 09:59:11

@dma_k: I don't know what I was drinking, but luckily didn't manage to mislead you.

Anton 2010-05-22 08:09:41

Answer 3

+1 A:

If you really want to do this in pure sed:

make libs-depends | sed 's/ /\n/g' | sed '/\.h$/!d;s/^/src\//' | sed 'G;/^\(.*\)\n.*\1/!h;$!d;${x;s/\n/ /g}'

The first sed command breaks the output up into separate lines, the second filters out everything but *.h and prepends 'src/', the third gloms the lines together without repetition.

Beta 2010-05-19 19:50:24

I am not an expert on `sed`. But I believe you can rephrase it with a single sed call, i.e.`make lib-depends | sed -e 's/ /\n/g' -e '/\.h$/!d;s/^/src\//' -e 'G;/^$.*$\n.*1/!h;$!d;${x;s/\n/ /g}'

Grzegorz Oledzki 2010-05-19 20:04:40

@GrzegorzOledzki: yes and no. The first call must be separate; I know I could combine the second and thirst, and maybe make things a little tighter elsewhere, but I was going for readability.

Beta 2010-05-19 21:47:57

@Beta: Thanks for the solution. Why the first sed should be separated? Because the 2nd sed needs to "sense" newly generated lines?

dma_k 2010-05-20 08:13:42

@dma_k: yes, exactly. It *can* be done with one call, but that's ugly.

Beta 2010-05-20 13:12:00

ansaurus

tags:

views:

answers:

sed script to remove file name duplicates

related questions