How do I push `sed` matches to the shell call in the replacement pattern?

Question

I need to replace several URLs in a text file with some content dependent on the URL itself. Let's say for simplicity it's the first line of the document at the URL.

What I'm trying is this:

sed "s/^URL=\(.*\)/TITLE=$(curl -s \1 | head -n 1)/" file.txt

This doesn't work, since \1 is not set. However, the shell is getting called. Can I somehow push the sed match variables to that subprocess?

Answer 1

So you are trying to call an external command from inside the replacement pattern of a sed substitution. I dont' think it can be done, the $... inside a pattern just allows you to use an already existent (constant) shell variable.

I'd go with Perl, see the /e option in the search-replace operator (s/.../.../e).

UPDATE: I was wrong, sed plays nicely with the shell, and it allows you do to that. But, then, the backlash in \1 should be escaped. Try instead:

sed "s/^URL=\(.*\)/TITLE=$(curl -s \\1 | head -n 1)/" file.txt

Answer 2

Try this:

sed "s/^URL=\(.*\)/\1/" file.txt | while read url; do sed "s@URL=\($url\)@TITLE=$(curl -s $url | head -n 1)@" file.txt; done

If there are duplicate URLs in the original file, then there will be n^2 of them in the output. The @ as a delimiter depends on the URLs not including that character.