How to make xargs execute in parallel but show complete output of a single instance at a time?
I am running some query commands in remote host with xargs in parallel which is working efficiently and very well but I have some problem in finding in which host I am getting the query from.
Right now the part of the script which is doing it, looks exactly like this:
export commands="cmd1; cmd2; "
hosts=("host1" "host2" "host3" "host4")
MaxInstance=10
echo_var(){ echo "Executing in $1:"; sleep 1; echo "Output of: $commands"; return 0;
}
export -f echo_var
printf "%s\n" ${hosts[@]} | xargs -n 2 -P $MaxInstance -I {} $(command -v bash) -c 'echo_var "$1"' _ {}which should output like the following because of sleep 1:
Executing in host1:
Executing in host2:
Executing in host3:
Executing in host4:
Output of: cmd1; cmd2;
Output of: cmd1; cmd2;
Output of: cmd1; cmd2;
Output of: cmd1; cmd2;To achieve the output that may look similar to the one shown at the end, I would have to edit the existing functions to the following to get closer to what I wanted. But even here if I use printf "%s\n%s" "$name" "$output" , it breaks sometimes at the new line. For example:
echo_var(){ name="Executing in $1:"; sleep 1; output="Output of: $commands"; echo -e "$name: $output" ## will work # echo -e "$name\n$output" ## will not work or even # printf "%s\n%s" "$name" "$output" ## will not work return 0;
}Without sacrificing the rapidness of parallel execution, I would like to have the output like the following. Is there any option in xargs to specify to show the output of the entire instance at a time?
Executing in host1:
Output of: cmd1; cmd2;
Executing in host2:
Output of: cmd1; cmd2;
Executing in host3:
Output of: cmd1; cmd2;
Executing in host4:
Output of: cmd1; cmd2; 1 Answer
Meet GNU Parallel (sudo apt install parallel):
GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially.
If you use xargs and tee today you will find GNU parallel very easy to use as GNU parallel is written to have the same options as xargs.
I can’t test it, but the command in your case should simply be:
parallel -n 2 -P $MaxInstance $(command -v bash) -c 'echo_var "$1"' _ {}Note that the moreutils package provides a parallel command which is different from GNU parallel.
Example run
This example runs echo {} start; sleep 1; echo {} ready for the arguments 1, 2 and 3, first with xargs and then with parallel:
$ echo -e '1\n2\n3' | xargs -n1 -P3 -I{} bash -c 'echo {} start; sleep 1; echo {} ready'
2 start
1 start
3 start
2 ready
1 ready
3 ready
$ parallel 'echo {} start; sleep 1; echo {} ready' ::: 1 2 3
1 start
1 ready
2 start
2 ready
3 start
3 readyOf course the echo | approach does work equally with parallel, I just like its argument list feature shown above – it does the exact same.
If you’re hesistant to use parallel because it’s not preinstalled, hard to install/update or anything like that I warmly recommend the program author’s blog article on Excuses for not installing GNU Parallel.
