progress information via pv for directory copy

I need to copy a very large directory (talking in terabytes here) and want to monitor the progress.

I found that pv is a nice utility, but how can I use it for copying directories recursively? (pv src > dst doesn’t work because they are directories)

PS: Running on OS X Mountain Lion, pv was installed from Mac Ports

Asked By: F.P

||

You could use tar or pax or cpio:

mkdir -p dst &&
  (cd src && tar cf - .) | pv -trb | (cd dst && tar xpf -)
Answered By: Stéphane Chazelas

Use rsync --progress [SRC] [DST]

Do check the man rsync page because it has a lot of very useful options. -a for archive is a good start, but it depends on your exact requirements.

Copying through a pipe will unnecessarily slow down the copy process, especially if it is files based.

Answered By: jippie
find source-dir -print0 | xargs -0 -I % pv % > dest-dir/%
Answered By: Christopher Hilla

Tar.

tar -cf - /var/log/ | pv | tar -C . -x

Example:

# tar -cf - /var/log/ | pv | tar -C . -x
tar: Removing leading `/' from member names
58MB 0:00:05 [ 2.2MB/s] [                   <=> 
Answered By: Anomalous Awe

You could do a du -b /directory/ on both source and destination while a normal copy command is in progress, and compare the two. This is just as effective and doesn’t slow the copy process by pushing it through a pipe.

Answered By: Apostle

Here are some commands to copy directories with progress information.


If there are many small files:

cp -av sourcedir targetdir | pv -l -s filecount > logfile

This will report progress based on number of files that are copied.

You can redirect to /dev/null if you don’t need logfile.

Use the following command to get filecount:

find sourcedir | wc -l

If there are few huge files:

tar c sourcedir | pv -s size | tar x -C targetdir

This will report progress based on bytes that are copied.

targetdir must exist.

Use the following command to get size:

du -sh sourcedir

If you want to use rsync:

rsync -ai sourcedir/ targetdir/ | pv -l -s filecount > logfile

Get filecount as shown above.

If you are copying on the same system then rsync -a is practically the same as cp -a . The advantages of rsync is when you are copying over the network or if you are updating (or comparing) a previous copy.

See here for more details:

Estimate time or work left to finish for rsync?

Compare directories but not content of files


for completeness some more alternatives

https://github.com/Xfennec/progress this tool can report progress even if the process already started.

https://github.com/aioobe/eta this can report progress based on any value obtained from a command. for example df -h

Answered By: Lesmana

I made a progress bar for rsync (in a wrapper):

rsyncy -a FROM/ TO

It looks like this:

rsyncy

More info on GitHub, install with pip3 install --user rsyncy

Answered By: laktak