Moving files without duplicating them to an S3FS-mounted storage bucket

Does the mv command (temporary) duplicate the moved files when the target is an S3FS mount?

I have a VM that is reaching total space consumption. So, I’d like to move some files to a storage bucket (mounted using S3FS) to free up some space.

Since I’m currently using 88.7% of 28.90GB, which is approximately 25.70GB and the directory I plan to move is 8.2GB, I’m afraid to use 100% and get stuck.

mv /path/sourcefolder/* /path/destinationfolder/
Asked By: Sig


s3fs uses local temporary storage to write objects to S3. -o max_dirty_data controls this amount and is set to 5 GB by default. When copying a larger file, e.g., 10 GB, s3fs will write the first 5 GB locally then write to S3. Afterwards it will write the second 5 GB to both then use a server-side copy to assemble the final object. Thus if you are worried about using too much local storage you should decrease this value. But doing so will trigger additional server-side copies, possibly slowing the overall write operation.

Answered By: Andrew Gaul

mv does not duplicate the files. cp does.

Answered By: someuser
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.