How to use "xargs" properly when argument list is too long

Question

Can someone please give me an example of using xargs on the below operation?

tar c $dir/temp/*.parse\
    | lzma -9 > $dir/backup/$(date '+%Y-%m-%d')-archive.tar.lzma

I get the error from bash "/bin/tar: Argument list too long"

Particularily I am trying to do LZMA compression on about 4,500 files; so this isn't surprising. I just don't know how to modify the above to use xargs and get rid of the error! Thanks.

Answer 1

Use find to pipe the 'wanted' filenames to a temporary file, and then use tar with the '-–files-from' command line option.

Edit:

or pipe them directly into each other to avoid the temporary file.

So: use find to list the wanted filenames | tar ... --files-from

Answer 2

Expanding on CristopheDs answer and assuming you're using bash:

tar c --files-from <(find $dir/temp -maxdepth 1 -name "*.parse") | lzma -9 > $dir/backup/$(date '+%Y-%m-%d')-archive.tar.lzma

The reason xargs doesn't help you here is that it will do multiple invocations until all arguments have been used. This won't help you here since that will create several tar archives, which you don't want.

Answer 3

Perhaps you want something like this?

find $dir/temp/ -name '*.parse' -print0 | tar --null -T - -c | lzma -9 > $dir/backup/$(date '+%Y-%m-%d')-archive.tar.lzma

Answer 4

Since you're on Linux, you have GNU tar, and you can use the '-F -' or '--files-from' option to read the file names.

However, you can also use:

--use-compress-program="lzma -9"

to specify the compression program to use, and simply give the compresses file name as the target file for the 'tar' command.

This was necessary for 'bzip2' compression before the '-j' option was added.

Answer 5

As a side note:

Always, always avoid xargs(1). It's a broken tool and is only remotely useful if you use it with the -0 option. Even then, it's almost always better to use find(1)'s -exec option, or a simple for or while loop.

Why is xargs so bad? Firstly, it splits input on whitespace, meaning all your filenames that contain whitespace will cause havoc. Secondly, it tries to be a smartass, and parse quotes for you. Which only leads to more headaches as you use it on filenames that contain quotes as part of the filename, such as songs: "I don't wanna miss a thing.mp3". This will just make xargs puke and curse at you that you quoted its input badly. No, you didn't: it should just learn that quotes do not belong in input data, they belong in shell scripts, and xargs has no business trying to parse them.

Mind you, xargs(1) doesn't do the whitespace splitting or the quote parsing when you pass -0 to it. It does the right thing, which is use NULL bytes to delimit filenames. But that means you need to give it input that uses NULL byte-delimited filenames (such as "find -foo -print0"). Which brings us back to: it's better to just use find's -exec: "find -foo -exec bar {} +".