Using Xargs For Repetition

last modified: November 11, 2010

CAUTION! Do not use xargs for repetition.

If you don't understand why xargs is BrokenAsDesigned, try the following code:

xargs <<EOF
xargs isn't broken
EOF

or, alternatively:

echo "xargs isn't broken" | xargs

Usual result is xargs stopping with diagnostic message:

xargs: unterminated quote

Always use xargs -0. This is the only way to avoid weird failures on untested corner cases. --BottomMind


Shell efficiency is often greatly impeded by fork/exec overhead. Consider the following code:

while
        read filename
do
        ls -ld $filename
done

The while loop makes it possible to invoke the same Shell command several times with the same general parameters, conveniently parameterized as shell variables. In this code, a fork and exec system call is performed for every file name in standard input. For loops with very simple commands, like ls in this case, the fork/exec overhead is the vast majority of the run time. It would be nice if ls could handle multiple files in a single invocation:

ls a b c

is the same as

ls a; ls b; ls c

a property true of many Shell commands (ls, cat, echo, etc.). But there is no syntactically convenient way to bunch up parameters coming through a pipeline. We could "bunch" parameters ourselves:

while
        read filename1 filename2 filename3 filename4
do
        ls -ld $filename1 $filename2 $filename3 $filename4
done

but it's difficult to handle the leftovers when the number of data items is not an even multiple of the number in the bunch (4 in this case).

Therefore:

Use xargs to "vectorize" loops. The following line replaces the loop above.

xargs ls -ld

Efficient shell scripts often resemble APL, they consist of vector operations.


CategoryUnixShellPattern


Loading...