Bash: Multiple for loops in Background

Is this the correct way to start multiple sequential processings in the background?

for i in {1..10}; do
    for j in {1..10}; do
        run_command $i $j;
    done &
done;

All j should be processed after each other for a given i, but all i should be processed simultaneously.

Asked By: Radio Controlled

||

The outer loop that you have is basically

for i in {1..10}; do
    some_compound_command &
done

This would start ten concurrent instances of some_compound_command in the background. They will be started as fast as possible, but not quite “all at the same time” (i.e. if some_compound_command takes very little time, then the first may well finish before the last one starts).

The fact that some_compound_command happens to be a loop is not important. This means that the code that you show is correct in that iterations of the inner j-loop will be running sequentially, but all instances of the inner loop (one per iteration of the outer i-loop) would be started concurrently.

The only thing to keep in mind is that each background job will be running in a subshell. This means that changes made to the environment (e.g. modifications to values of shell variables, changes of current working directory with cd, etc.) in one instance of the inner loop will not be visible outside of that particular background job.

What you may want to add is a wait statement after your loop, just to wait for all background jobs to actually finish, at least before the script terminates:

for i in {1..10}; do
    for j in {1..10}; do
        run_command "$i" "$j"
    done &
done

wait
Answered By: Kusalananda

If you have GNU Parallel you would do:

parallel -j0 'for j in {1..10}; do run_command {} $j; done' ::: {1..10}

One of the benefits is that output from parallel running run_commands will not mix.

Answered By: Ole Tange