Recurring movement of complete files

I would like to address a query that may have been previously discussed or addressed indirectly through other inquiries.

I am in need of a solution for recurrently moving files with a specific file extension from one folder to another. However, due to the automated extraction process, it is imperative that the script can identify and skip files that are not completely extracted.

I have already devised a preliminary script and tested it with crontab. Now, I seek to enhance it by incorporating the following functionalities:

  • Specifying the handling of files with a specific extension
  • Implementing a mechanism to skip files that are incompletely extracted

Best
Luca


After the first reply, I made this script.
The file before extraction are .rar (also divided in .part), but with this script I almost resolved everything

#!/bin/bash

# Source folder
source_folder="-----"

# Destination folder
destination_folder="-----"

# File extension to be moved file_extension="-----"

# Iteration through the files with the specified extension
for file in "$source_folder"/*"$file_extension"; do
    # Check if the file is open
    if ! lsof "$file" >/dev/null 2>&1; then
        # move the file
        mv "$file" "$destination_folder"
    fi
done
Asked By: Luca Panareo

||

You can use a utility such as fuser or lsof to find whether a process is writing to the file. This is only useful if the files are extracted by a process on the same host, and if that process continuously has the file open for writing.

What to do if the extraction process fails or aborts (e.g. due to a system reboot)? What do you want to do with an incompletely extracted file that isn’t actively being extracted?

Can you recognize the end of a completely extracted file? If you can, then I would use that as your criterion. Depending on the file format, I’d use awk, perl or python to detect the end of file. (If it’s in a structured format, such as JSON, you want to use a library to parse it; such libraries are readily available for Perl and Python.)

Once you have a script, say, is-complete, that decides whether a file is complete or not (by returning exit code 0 for success and any other number for failure), you can move the files in a single command:

find extraction-directory -type f -name '*.desiredext' -exec is-complete {} ; -exec mv -t final-directory/ {} ;
Answered By: Reinier Post
Categories: Answers Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.