GNU parallel: why does diagnostic output look like sequential execution rather than parallel execution?

Scenario:

$ cat libs.txt
lib.a
lib1.a

$ cat t1a.sh
f1()
{
        local lib=$1
        stdbuf -o0 printf "job for $lib startedn"
        sleep 2
        stdbuf -o0 printf "job for $lib donen"
}
export -f f1
cat libs.txt | SHELL=$(type -p bash) parallel --jobs 2 f1

Invocation and output:

$ time bash t1a.sh
job for lib.a started
job for lib.a done
job for lib1.a started
job for lib1.a done

real    0m2.129s
user    0m0.117s
sys     0m0.033s

Here we see that execution of f1 was indeed in parallel (real 0m2.129s).

However, diagnostic output looks like execution was sequential.

I expected the following diagnostic output:

job for lib.a started
job for lib1.a started
job for lib.a done
job for lib1.a done

Why does diagnostic output look like sequential execution rather than parallel execution?

How to fix the diagnostic output so that it does look like parallel execution?

Asked By: pmor

||

From the man pages of GNU parallel:

--group

Group output.

Output from each job is grouped together and is only printed when the
command is finished.
Stdout (standard output) first followed by stderr
(standard error).

This takes in the order of 0.5ms CPU time per job and depends on the
speed of your disk for larger output.

--group is the default.

See also: --line-buffer --ungroup --tag

[…]

--line-buffer

--lb

Buffer output on line basis.

--group will keep the output together for a whole job. --ungroup allows output to mixup with half a line coming from one job and half a
line coming from another job. --line-buffer fits between these two:
GNU parallel will print a full line, but will allow for mixing lines
of different jobs.

So you should add either --line-buffer or --ungroup to your parallel command (according to your preferred behavior):

$ grep parallel t1a.sh 
cat libs.txt | SHELL=$(type -p bash) parallel --line-buffer --jobs 2 f1

$ bash t1a.sh 
job for lib.a started
job for lib1.a started
job for lib.a done
job for lib1.a done
Answered By: aviro
Categories: Answers Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.