pv (progress bar) and gzip

Why is this not possible?

pv ${dest_file} | gzip -1

pv is a progress bar

error

gzip: compressed data not written to a terminal. Use -f to force compression.
For help, type: gzip -h
   0 B 0:00:00 [   0 B/s] [>                                   ]  0%

This works

pv ${file_in} | tar -Jxf - -C /outdir
Asked By: clarkk

||

What are you trying to achieve is to see the progress bar of the compression process. But it is not possible using pv. It shows only transfer progress, which you can achieve by something like this (anyway, it is the first link in the google):

pv input_file | gzip > compressed_file

The progress bar will run fast, and then it will wait for compression, which is not observable anymore using pv.

But you can do that other way round and watch the output stream, bot here you will not be able to see the actual progress, because pv does not know the actual size of the compressed file:

gzip <input_file | pv > compressed_file

The best I found so far is the one from commandlinefu even with rate limiting and compression of directories:

$D=directory
tar pcf - $D | pv -s $(du -sb $D | awk '{print $1}') --rate-limit 500k | gzip > target.tar.gz
Answered By: Jakuje

Yes, it is kinda possible using the lines of the file, not the bytes.
You get an accurate enough progress bar:

cat input_file | pv -ls $( wc -l input_file ) | gzip -c -- > compressed_file
Answered By: Archangel

If available, you can use 7z instead to get a percentage progress in a super simple command:

7z a spaCy.tar.gz spaCy.tar 

Example output:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (906E9),ASM,AES-NI)

Open archive: spaCy.tar.gz
--
Path = spaCy.tar.gz
Type = gzip
Headers Size = 20

Scanning the drive:
1 file, 326451200 bytes (312 MiB)

Updating archive: spaCy.tar.gz

Items to compress: 1

 32% U spaCy.tar   

         
Answered By: qwr
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.