trying to CP a directory to /dev/null

What I’m really tring to do, it verify the the data on an external drive is still readable. I thought the simplest and surest way to do this would be to copy everything to /dev/null. It works great for a file. But when I add -R to recursive copy a dirfectory structure, if fails (presumably) because it can’t create subdirectories in /dev/null.

When I try :

cp -R Flintstones/ /dev/null

I get :

cp: cannot overwrite non-directory '/dev/null' with directory 'Flintstones/'

But this works:

cd Flintstones
cp Fred.txt /dev/null

How can I copy the entire directory in one go??

Thank you, and please be gentle with the newbie.

Asked By: pastorbadger

||

Use cat instead of cp

Instead of using cp to read every file, you can use the cat command.

The screen below illustrates catall.sh bash script. The first section shows results for a short run (40 minutes!) on /home/rick/. The second section shows in progress display on /:

catall progress display.gif

On my system there are many partitions (including two full Windows 10 installations) where all the files are being read.

catall.sh bash script

The bash script must be called with sudo powers because there are files in Linux a regular user cannot read. Additionally you must pass a parameter to the starting location to begin reading files, eg / or /mnt/ext_drive, etc. :

#!/bin/bash

# NAME: catall.sh (cat every single file)
# PATH: $HOME/askubuntu/
# DESC: Answer for: https://askubuntu.com/questions/1184103/trying-to-cp-a-directory-to-dev-null/1184105#1184105
# DATE: October 27, 2019.

[[ $(id -u) != 0 ]] && { echo "Must be called with sudo" >&2 ; exit 2 ; }
[[ $1 != /* ]] && { echo "Parameter 1 must be a path" >&2 ; exit 3 ; }


DirCnt=0    # Directory count
FileCnt=0   # File
LinkCnt=0   # Symbolic Link
BdevCnt=0   # Block Device
CdevCnt=0   # Character Device
PipeCnt=0   # Pipe
SockCnt=0   # Socket
ZeroCnt=0   # Zero sized files count
SkipCnt=0   # Files skipped

updatedb    # Update locate's database for files added today
StartSec=$SECONDS

while read File ; do
    if [[ ! -e "$File" ]] ; then : # File no longer exists, stale database
    elif [[ -d "$File" ]] ; then (( DirCnt++ ))
    elif [[ -h "$File" ]] ; then (( LinkCnt++ ))
    elif [[ -b "$File" ]] ; then (( BdevCnt++ ))
    elif [[ -b "$File" ]] ; then (( CdevCnt++ ))
    elif [[ -c "$File" ]] ; then (( CdevCnt++ ))
    elif [[ -p "$File" ]] ; then (( PipeCnt++ ))
    elif [[ -S "$File" ]] ; then (( SockCnt++ ))
    # elif [[ $File == /mnt/* ]] ; then (( SkipCnt ++ ))
    elif [[ -s "$File" ]] ; then
        cat "$File" 1>/dev/null
        (( FileCnt++ ))
    else
        (( ZeroCnt++ ))
    fi
    printf "Number of Directories: %'d Files: %'d r" "$DirCnt" "$FileCnt"
done <<<"$(locate "$1")"

echo # Line feed to keep statistics visible when job ends
printf "Number of Sybolic Links: %'dn"     "$LinkCnt"
printf "Number of Block Devices: %'dn"     "$BdevCnt"
printf "Number of Character Devices: %'dn" "$CdevCnt"
printf "Number of Pipes: %'dn"             "$PipeCnt"
printf "Number of Sockets: %'dn"           "$SockCnt"
printf "Number of Zero sized files: %'dn"  "$ZeroCnt"
# printf "Number of Skipped files: %'dn"     "$SkipCnt"
echo
EndSec=$SECONDS
TotalSec=$(( EndSec - StartSec ))
printf "Total Seconds: %'dn"               "$TotalSec"

Two lines are commented out for skipping specific directories:

    # elif [[ $File == /mnt/* ]] ; then (( SkipCnt ++ ))
# printf "Number of Skipped files: %'dn"     "$SkipCnt"

Remove the comment tag (#) and change /mnt/* to the directory you wish to skip to save time. In my case /mnt/ represents over a million files in Windows and other Ubuntu installations on different partitions.


Speed of cat is terrible

The final results were:

$ sudo ./catall.sh /

Number of Directories: 378,571 Files: 1,741,640 
Number of Sybolic Links: 92,011
Number of Block Devices: 25
Number of Character Devices: 55
Number of Pipes: 48
Number of Sockets: 138
Number of Zero sized files: 210,515

Total Seconds: 82,554

The process took 23 hours or almost a full day!

Much better speed can be achieved with dd even though a full partition is read including spaces where no files exist:

$ time sudo dd if=/dev/nvme0n1p4 of=/dev/null bs=100M
3719+1 records in
3719+1 records out
389969573888 bytes (390 GB, 363 GiB) copied, 249.867 s, 1.6 GB/s

real    4m9.896s
user    0m0.017s
sys     2m38.305s

Why use sudo with catall.sh script?

This example illustrates why sudo is required:

$ locate udev.log.1.gz
/mnt/clone/var/log/upstart/udev.log.1.gz
/var/log/upstart/udev.log.1.gz

$ cat /var/log/upstart/udev.log.1.gz | wc
cat: /var/log/upstart/udev.log.1.gz: Permission denied
      0       0       0

$ sudo cat /var/log/upstart/udev.log.1.gz | wc
      0       1      82
Answered By: WinEunuuchs2Unix

Hello and welcome to Ask Ubuntu!

If I understood your question correctly you just want to run some command that will read all files in the given directory, but you are not interested in any output other than potential error messages saying files could not be read. You can achieve that for example using the grep command, like this:

grep -r Zaphod Flintstones/

The above command means that grep will recursively go through all files in the Flintstones directory looking for the string “Zaphod” (you could put some other string there, it does not matter, just something for grep to search for, if you want no output pick something that you think will not occur in any file). If the command completes without error messages about unreadable files, then that means the directory is readable.

Not saying this is the best way, probably there are more elegant ways, but the above is one way of doing it.

Answered By: Elias

I found a nice way to replace rsync for reading out my harddisk without copying.

In bash window #1 run this command:

tar -cv <path to your hard disk> 2>/tmp/tt |pv >/dev/null

Progress in Bytes / MB / GB is shown along with the current transfer speed, e.g.:

$ sudo tar -cv /mnt/mnt/ 2>/tmp/tt |pv >/dev/null 
111GiB 0:21:57 [99,0MiB/s] [                 <=>                          ]

In bash window #2 run this command to show the file that is currently being read (terminate with ctrl-c once the test is completed).

tail -f /tmp/tt

This works even if your hard disk contains special files like block device descriptors.

Background:
tar -cv <dirname>
creates an archive containing all the files in <dirname>. The output goes through the pipeline ("|"), the filenames are printed to stderr which is redirected to /tmp/tt.
pv prints out the size and the current transfer speed. Using the option "-s" you could tell pv how large your data is (e.g. "pv -s 370G"), so it can also print out the percentage of data already copied:

$ sudo tar -cv /media/jcs/SG4TBR1/ 2>/tmp/tt |pv -s370G >/dev/null 
24,7GiB 0:04:07 [ 130MiB/s] [=>                   ]  6% ETA 0:57:27

"tail -f /tmp/tt" simply prints the contents of /tmp/tt and continues to show any changes (-f: "follow").

Answered By: jcsjcs

One possible solution I came up with to ensure all of the content of all files can be read is this right here:

sudo find -type f -exec cat {} + > /dev/null

sudo so all files will be read and none of the errors will be related to permissions, find to get a list of files, -type f to filter out directories (since cat prints an error for those), and -exec cat {} + > /dev/null to tell cat to cat all of the files in find’s current batch to /dev/null. This works fairly well, but I haven’t tested with block, pipe, or other files, and you don’t get any progress either.

P.S. This was me mostly checking to see if a certain drive was still good.

Answered By: BrainStorm.exe

Building on what WinEunuuchs2Unix wrote, I prettied it up a bit:

#!/bin/bash
# NAME: cachewarmup.sh
# DESC: Adapted from https://askubuntu.com/questions/1184103/trying-to-cp-a-directory-to-dev-null/1184105#1184105
# DATE: 25 Feb 2023

[[ $(id -u) != 0 ]] && { echo "Must be called with sudo" >&2 ; exit 2 ; }
[[ $1 != /* ]] && { echo "Input flag must be the starting directory ( sudo cachewarmup / for root directory )" >&2 ; exit 3 ; }

# NOTE: Put Omitted Paths below:
# =============================
OP0="/mnt/**/*"
OP1="/media/$USER/**/*"
OP2=""
OP3=""
OP4=""
OP5=""
OP6=""
OP7=""
OP8=""
OP9=""
# =============================

SkipCnt=0                                                                               # Count of skipped files
StalCnt=0                                                                               # Count of files that existed last run, but don't now
ZeroCnt=0                                                                               # Count of zero sized files
DirCnt=0                                                                                # Count of directories
LinkCnt=0                                                                               # Count of symbolic links
CdevCnt=0                                                                               # Count of character devices
BdevCnt=0                                                                               # Count of block devices
PipeCnt=0                                                                               # Count of pipes
SockCnt=0                                                                               # Count of sockets
FileCnt=0                                                                               # Count of files cache-loaded

StartSec=$SECONDS
clear
echo "Updating Locate's database... please wait..."
updatedb > /dev/null 2>&1                                                               # Update locate's database for recently added files
clear
tput civis
for file in $(find $1 -maxdepth 10000 -xdev -ignore_readdir_race); do
    if [[ "$file" =~ ^($OP0|$OP1|$OP2|$OP3|$OP4|$OP5|$OP6|$OP7|$OP8|$OP9)$ ]]; then (( SkipCnt++ )) # Count of skipped files
    elif [[ ! -e "$file" ]] ; then (( StalCnt++ ))                                      # Count of files that existed last run, but don't now
    elif [[ ! -s "$file" ]] ; then (( ZeroCnt++ ))                                      # Count of zero sized files
    elif [[ -d "$file" ]] ; then (( DirCnt++ ))                                         # Count of directories
    elif [[ -h "$file" || -L "$file" ]] ; then (( LinkCnt++ ))                          # Count of symbolic links
    elif [[ -c "$file" ]] ; then (( CdevCnt++ ))                                        # Count of character devices
    elif [[ -b "$file" ]] ; then (( BdevCnt++ ))                                        # Count of block devices
    elif [[ -p "$file" ]] ; then (( PipeCnt++ ))                                        # Count of pipes
    elif [[ -S "$file" ]] ; then (( SockCnt++ ))                                        # Count of sockets
    elif [[ -f "$file" && -s "$file" ]] ; then                                          # File must exist, and not be any of the above.
        cat "$file" 1>/dev/null
        (( FileCnt++ ))                                                                 # Count of files cache-loaded
    else
        (( SkipCnt++ ))                                                                 # Count of any files not otherwise processed.
    fi
    printf "  Directories:      %dn"       "$DirCnt"
    printf "  Cached Files:     %dn"       "$FileCnt"
    printf "  Symbolic Links:   %dn"       "$LinkCnt"
    printf "  Block Devices:    %dn"       "$BdevCnt"
    printf "  Character Devices:    %dn"   "$CdevCnt"
    printf "  Pipes:        %dn"           "$PipeCnt"
    printf "  Sockets:      %dn"           "$SockCnt"
    printf "  Zero Sized files: %dn"       "$ZeroCnt"
    printf "  Skipped files:    %dn"       "$SkipCnt"
    printf "  Stale files:      %dn"       "$StalCnt"
    echo
    tput cup 11 1;tput el;printf "%s" "$file"
    tput cup 12 0;tput el
    tput cup 13 0;tput el
    Elapsed=($SECONDS - $StartSec)
    tput cup 14 1;tput el;printf "Time %dh:%dm:%ds" "$((($Elapsed)/3600))" "$(((($Elapsed)%3600)/60))" "$((($Elapsed)%60))"
    tput cup 0 0
done <<<"$(locate "$1")"
tput cnorm

If you want to nice it up:

nice gnome-terminal -- /bin/sh -c 'echo Warming up ZFS ARC...; sleep 60; sudo cachewarmup /; sleep 300' 19

That’ll run from a keyboard shortcut or the Startup Applications.

Answered By: Anon
Categories: Answers Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.