Moving files without duplicating them to an S3FS-mounted storage bucket
Does the mv command (temporary) duplicate the moved files when the target is an S3FS mount?
I have a VM that is reaching total space consumption. So, I’d like to move some files to a storage bucket (mounted using S3FS) to free up some space.
Since I’m currently using 88.7% of 28.90GB, which is approximately 25.70GB and the directory I plan to move is 8.2GB, I’m afraid to use 100% and get stuck.
mv /path/sourcefolder/* /path/destinationfolder/
s3fs uses local temporary storage to write objects to S3. -o max_dirty_data controls this amount and is set to 5 GB by default. When copying a larger file, e.g., 10 GB, s3fs will write the first 5 GB locally then write to S3. Afterwards it will write the second 5 GB to both then use a server-side copy to assemble the final object. Thus if you are worried about using too much local storage you should decrease this value. But doing so will trigger additional server-side copies, possibly slowing the overall write operation.
mv does not duplicate the files. cp does.