Merging folders with mv?

If I use mv to move a folder called “folder” to a directory that already contains “folder” will they merge or will it be replaced?

Asked By: Dominique

||
Source

mv cannot merge or overwrite directories, it will fail with the message "mv: cannot move ‘a’ to ‘b’: Directory not empty", even when you’re using the --force option.

You can work around this using other tools (like rsync, find, or even cp), but you need to carefully consider the implications:

  • rsync can merge the contents of one directory into another (ideally with the --remove-source-files1 option to safely delete only those source files that were transferred successfully, and with the usual permission/ownership/time preservation option -a if you wish)
    … but this is a full copy operation, and can therefore be very disk-intensive.
  • You can use find to sequentially recreate the source directory structure at the target, then individually move the actual files
    … but this has to recurse through the source multiple times and can encounter race conditions (new directories being created at the source during the multi-step process)
  • cp can create hard links (simply put, additional pointers to the same existing file), which creates a result very similar to a merging mv (and is very IO-efficient since only pointers are created and no actual data has to be copied)
    … but this again suffers from a possible race condition (new files at the source being deleted even though they weren’t copied in the previous step)
  • You can combine rsync‘s --link-dest=DIR option (to create hardlinks instead of copying file contents, where possible) and --remove-source-files to get a semantic very similar to a regular mv.
    For this, --link-dest needs to be given an absolute path to the source directory (or a relative path from the destination to the source).
    … but this is using --link-dest in an unintended way (which may or may not cause complications), requires knowing (or determining) the absolute path to the source (as an argument to --link-dest), and again leaves an empty directory structure to be cleaned up as per 1.     (Note: This won’t work anymore as of rsync version 3.2.6)

Which of these workarounds (if any) is appropriate will very much depend on your specific use case.
As always, think before you execute any of these commands, and have backups.

1: Note that rsync --remove-source-files won’t delete any directories, so you will have to do something like find -depth -type d -empty -delete afterwards to get rid of the empty source directory tree.

Answered By: n.st
rsync -av /source/ /destination/
(after checking)
rm -rf /source/
Answered By: fazie

One way to accomplish this would be to use:

mv folder/* directory/folder/
rmdir folder

As long as no two files have the same name in folder and directory/folder, you will achieve the same result i.e. merging.

Answered By: asheeshr

Here is a way that will merge the directories. It is much faster than rsync since it just renames the files instead of copying them and then deleting them.

cd source; find -type f -print0 | xargs -0 -n 1 -I {} mv '{}' 'dest/{}'
Answered By: Jewel

I’d recommend these four steps:

cd "${SOURCE}"; 
find . -type d -exec mkdir -p "${DEST}/{}" ; 
find . -type f -exec mv {} "${DEST}/{}" ; 
find . -type d -empty -delete

or better yet, here’s a script that implements semantics similar to mv:

#!/bin/bash

DEST="${@:${#@}}"

for SRC in "${@:1:$((${#@} -1))}"; do   (
    cd "$SRC";
    find . -type d -exec mkdir -p "${DEST}"/{} ;
    find . -type f -exec mv {} "${DEST}"/{} ;
    find . -type d -empty -delete
) done

The quotes around the SRC and DEST variables will preserve whitespace in path names.

Answered By: Jonathan Mayer

Here is a script that worked for me. I prefer mv over rsync, so I use Jewel and Jonathan Mayer’s solutions.

#!/bin/bash

# usage source1 .. sourceN dest

length=$(($#-1))
sources=${@:1:$length}
DEST=$(readlink -f ${!#})
for SRC in "$sources"; do
    pushd "$SRC";
    find . -type d -exec mkdir -p "${DEST}/{}" ;
    find . -type f -exec mv {} "${DEST}/{}" ;
    find . -type d -empty -delete
    popd
done
Answered By: xer0x

You can use the -l option of the cp command, which creates hard links of files on the same filesystem instead of full-data copies. The following command copies the folder source/folder to a parent folder (destination) which already contains a directory with the name folder.

cp -rl source/folder destination
rm -r source/folder

You may also want to use the -P (--no-dereference – do not de-reference symbolic links) or -a (--archive – preserve all metadata, also includes -P option), depending on your needs.

Answered By: palswim

It is not a good idea to use commands like cp or rsync. For large files, it will take a long time. mv is much faster since it only update the inodes without copying the files physically. A better option is to use the file manager of your operating system. For Opensuse, there is a file manager called Konquerer. It can move files without actually copying them. It has “cut and paste” function like in Windows. Just select all the sub-directories in directory A. Right click and “move into” directory B which may contain sub-directories with the same names. It will merge them. There are also options whether you want to overwrite or rename files with the same name.

Answered By: simon

For the purest copies, I use the tar (-)B blockread copy method.

example, from within source path (‘cd’ there if necessary):

tar cBf - <sourcefolder> | (cd /your/target/folder ; tar xBf -)

this creates an exact copy of the source tree, WITH the owner and permissions intact. And if the target folder exists, the data will be merged. Only files that are already existing will be overwritten.

Example:

 $ cd /data1/home
 $ tar cBf - jdoe | (cd /data2/home ; tar xBf -)

When the copy action is successful, you can remove the source (rm -rf <source>). Of course this is not an exact move: the data will be copied, until you remove the source.

As option you can be verbose (display on screen the file being copied), with -v: tar cBvf -

  • c: create
  • B: read full block (for pipe read)
  • v: verbose
  • f: file to write
  • x: extract
  • -: stdout/stdin

sourcefolder can also be * (for anything in current folder)

Answered By: gjs

Quick Python solution that only walks the source file tree once

Since I could not find a satisfactory pre-existing solution, I decided to make a quick Python script to achieve it.

In particular, this method is efficient because it only walks the source file tree once bottom up.

It will also allow you to quickly tweak things like file overwrite handling to your liking.

Usage:

move-merge-dirs src/ dest/

will move all contents of src/* into dest/ and src/ will disappear.

move-merge-dirs

#!/usr/bin/env python3

import argparse
import os

def move_merge_dirs(source_root, dest_root):
    for path, dirs, files in os.walk(source_root, topdown=False):
        dest_dir = os.path.join(
            dest_root,
            os.path.relpath(path, source_root)
        )
        if not os.path.exists(dest_dir):
            os.makedirs(dest_dir)
        for filename in files:
            os.rename(
                os.path.join(path, filename),
                os.path.join(dest_dir, filename)
            )
        for dirname in dirs:
            os.rmdir(os.path.join(path, dirname))
    os.rmdir(source_root)

if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description='Move merge src/* into dest. Overwrite existing files.'
    )
    parser.add_argument('src_dir')
    parser.add_argument('dest_dir')
    args = parser.parse_args()
    move_merge_dirs(args.src_dir, args.dest_dir)

GitHub upstream.

See also: https://stackoverflow.com/questions/22588225/how-do-you-merge-two-directories-or-move-with-replace-from-the-windows-command

Tested on Python 3.7, Ubuntu 18.04.

Answered By: Ciro Santilli OurBigBook.com

This is the command for moving files & folders to other destination:

$ mv /source/path/folder /target/destination/

Remember: mv command will not work if the folder is b̲e̲i̲n̲g m̲e̲r̲ge̲d̲ (i.e. another folder with the same name already exist in the destination) and the d̲e̲s̲t̲i̲n̲a̲t̲i̲o̲n̲ o̲n̲e̲ i̲s̲ n̲o̲t̲ e̲m̲pt̲y.

mv: cannot move ‘/source/path/folder’
to ‘/target/destination/folder’: Directory not empty

If the destination folder is empty, the above command will work fine.

So, in order to merge both folders in any case,
Either do it in 2 commands:

$ cp -rf /source/path/folder /target/destination/
$ rm -rf /source/path/folder

Or combine both as a one-time command:

$ cp -rf /source/path/folder /target/destination/ && rm -rf /source/path/folder

mv = move
cp = copy
rm = remove

-r for directory (folder)
-f force execution

Answered By: TalESid

No sense copying folders that are empty – YMMV

#!/bin/bash

# usage source1 .. sourceN dest

length=$(($#-1))
sources=${@:1:$length}
DEST=$(readlink -f ${!#})
for SRC in $sources; do
    pushd "$SRC";
    # Only one scan - we only need folders with files
    find . -type f | while read FILE ; do
        DIRNAME=`dirname "$FILE"`
        # Create the lowest level directory at once
        if [ ! -d "$DEST/$DIRNAME" ] ; then
            mkdir -v "$DEST/$DIRNAME"
        fi
        mv -v "$FILE" "$DEST/$FILE"
    done
    # Remove the directories no longer needed
    find . -type -d | sort -r | xargs -i rmdir "{}"
    popd
done
  • find not executed multiple times
  • mkdir -p is executed even after finding directories sequentially
Answered By: A Concerned Linux Denizen

You can merge a and b with:

shopt -s dotglob
mv a/* b

Before mv:

.
├── a
│   ├── c
│   │   └── x
│   └── .x
└── b
    ├── y
    └── .y

After mv:

.
├── a
└── b
    ├── c
    │   └── x
    ├── .x
    ├── y
    └── .y

dotglob allows you to move hidden dot files like .x

Use rmdir to remove empty directory.

Answered By: sir__finley

Use mv with find. You can do this in one pass.

cd "$SRC"
find -type d -exec mkdir -vp "$DST"/{} ; -or -exec mv -nv {} "$DST"/{} ;

… where $SRC and $DST are the source and destination directories, respectively.

Explanation

  • -type d tests if the item is a directory. If it is a directory, we proceed to the next action or test: -exec ….
  • In -exec … {} ;, the {} is replaced with the path to the current item, relative to the current working directory. The ; indicates the end of this -exec … command.
  • In mkdir -pv …, -pv is equivalent to -p -v. The -p means to create all intermediate directories, as needed, and not raise an error if the directory already exists. The -v means --verbose and just tells it to print a message for each directory created, so you can see what it is doing. "$DST"/{} will be expanded to the destination directory, including all needed quotes.
  • The -or is the interesting part, which allows us to do this in one pass. With the find command, every test (e.g., -type d) or action (e.g., -exec …) result in a status of true or false, depending on if the test passed or action succeeded. Tests and actions can be connected using -and, -or, -not, -true, -false, and ( … ). When you add multiple tests and/or actions without an explicit boolean operator, they are implicitly AND’d together. Thus, the above command is equivalent to this: find ( -type d -and -exec mkdir -vp "$DST"/{} ; ) -or -exec mv -nv {} "$DST"/{} ;. Thus, if -type d passes, then it goes on to the next action (-exec …). If not, then that first branch of the -or is false, and it goes to the second branch, which covers anything that is not a directory (e.g., files).
  • In mv -nv {} "$DST"/{}, -nv is equivalent to -n -v. The -n tells it to not overwrite any files in the destination directory.. The -v tells it to report a message for every file moved, so you can see what it is doing.
  • Directories will be created before their files are moved. find uses breadth-first traversal by default.
  • The {} does NOT need to be enclosed in quotes, even if the item it stands for includes spaces.
  • Empty directories at the source will remain.

Example

If you wanted to copy /usr/local into /usr, you could enter it like this.

cd /usr/local
find -type d -exec mkdir -vp ../{} ; -or -exec mv -nv {} ../{} ;

It would result in commands like this:

mkdir -pv .././bin
mv -nv ./bin/pip .././bin/pip
mv -nv ./bin/pip3 .././bin/pip3
mv -nv ./bin/python3 .././bin/python3
mv -nv ./bin/python3 .././bin/python3
mv -nv ./bin/xxhsum .././bin/xxhsum
mkdir -pv .././etc
mkdir -pv .././include
mv -nv ./include/xxh3.h .././include/xxh3.h
mv -nv ./include/xxhash.h .././include/xxhash.h

… and so on

How to preview

To see what commands will be run, add echo before each command, right after the -exec, like this:

cd "$SRC"
find -type d -exec echo mkdir -vp "$DST"/{} ; -or -exec echo mv -nv {} "$DST"/{} ;
                   ‾‾‾‾                               ‾‾‾‾
Answered By: Alex Quinn

Just wanted to share my solution to a similar issue. I’ve got a mess. Multiple copies of a directory where each copy has edits to files, and basically just need to merge it back together:

  • without losing newer changes.
  • without losing files by ones that may have been corrupted.
  • without unnecessary copy operations.
  • retain as much about files as possible (extended attributes, ACLs, etc)
  • also, to reduce disk usage by hard linking files that have been copied several times, have totally different filenames, and may exist in any number of directories.

That last operation will be anther step where I build a list of duplicate files in those directories based on inode/size/md5sum comparisons, and then decide whether to hard-link or simply delete duplicates (how to control which one to save I have yet to decide).

However, I plan to complete my first operations with the following:

# hard-link over any files that don't exist in the destination while removing them from source
rsync -livrHAX --remove-source-files --ignore-existing --link-dest=../src/ src/ dst/

# move over existing files that are newer and remove them from source, keeping backup of ones that were replaced
# (after verifying during test drills that inodes of moved files are the same, I conclude that this doesn't slow-copy files, but YMMV or check rsync source for your OS/arch/filesystem)
rsync -buvaiHAX --remove-source-files --suffix=".bak_older" src/ dst/

# move over the rest of the files that are older than the ones in the destination, remove them from source, and retain backups of ones that were replaced
rsync -bvaiHAX --remove-source-files --suffix=".bak_newer" src/ dst/

# remove empty directories recursively
find src -type d -exec rmdir -p "{}" ; 2>/dev/null 

# src/ should hopefully now no longer exist

# check for averted clobbers against older files to manually verify that the replacements are acceptable
find dst -name '*.bak_older'

# check for averted clobbers again newer files to manually verify that the replacements aren't out-dated (in terms of whatever is important to you)
find dst -name '*.bak_older'

I’ll report back to this post if I have any major updates to my procedures, but this in effect feels like a quick & safe "directory merge" operation

Answered By: Brandon

The above answers are good, but doing this process made me nervous. I wanted to share a test script that demonstrates what the rsync method will actually do.

    reset_rsync_test_local_move(){
        LOCAL_DPATH=$HOME/tmp/rsync-test/local
        MOVE_TEST_ROOT=$LOCAL_DPATH/rsync_move_test
        # Setup home data
        echo "LOCAL_DPATH = $LOCAL_DPATH"
        if [ -d "$LOCAL_DPATH" ]; then
            rm -rf $LOCAL_DPATH
        fi
        mkdir -p $LOCAL_DPATH
        mkdir -p $MOVE_TEST_ROOT
        # Pretend that we accidently botched a move and have a repo inside of a repo
        # so the goal is merge all files from repo/repo into repo
        mkdir -p $MOVE_TEST_ROOT/repo/
        mkdir -p $MOVE_TEST_ROOT/repo/primes/
        mkdir -p $MOVE_TEST_ROOT/repo/perfect/
        mkdir -p $MOVE_TEST_ROOT/repo/nat/
    
        mkdir -p $MOVE_TEST_ROOT/repo/repo
        mkdir -p $MOVE_TEST_ROOT/repo/repo/primes/
        mkdir -p $MOVE_TEST_ROOT/repo/repo/perfect/
        mkdir -p $MOVE_TEST_ROOT/repo/repo/nat/
    
        # Some of the primes ended up in the correct and the botched repo
        touch $MOVE_TEST_ROOT/repo/primes/prime02
        touch $MOVE_TEST_ROOT/repo/primes/prime05
        touch $MOVE_TEST_ROOT/repo/primes/prime13
        touch $MOVE_TEST_ROOT/repo/repo/primes/prime03
        touch $MOVE_TEST_ROOT/repo/repo/primes/prime11
        # For prime7, lets say there is a conflict in the data contained in the file
        echo "correct data" > $MOVE_TEST_ROOT/repo/primes/prime07
        echo "botched data" > $MOVE_TEST_ROOT/repo/repo/primes/prime07
    
        # All of the perfects ended up in the botched repo
        touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect006
        touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect028
        touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect496
    
        # The naturals have some symlinks, so we need to be careful there
        touch $MOVE_TEST_ROOT/repo/nat/nat04
        touch $MOVE_TEST_ROOT/repo/nat/nat06
    
        # basedir nats
        touch $MOVE_TEST_ROOT/repo/nat/nat01
        ln -s $MOVE_TEST_ROOT/repo/primes/prime02 $MOVE_TEST_ROOT/repo/nat/nat02
        (cd $MOVE_TEST_ROOT/repo/nat/ && ln -s ../primes/prime05 nat05)
        ln -s $MOVE_TEST_ROOT/repo/primes/prime11 $MOVE_TEST_ROOT/repo/nat/nat11
    
        # Botched nats
        touch  $MOVE_TEST_ROOT/repo/repo/nat/nat08
        ln -s $MOVE_TEST_ROOT/repo/primes/prime07  $MOVE_TEST_ROOT/repo/repo/nat/nat07 
        (cd $MOVE_TEST_ROOT/repo/repo/nat/ && ln -s  ../primes/prime03 nat03)
        ln -s $MOVE_TEST_ROOT/repo/repo/primes/prime11 $MOVE_TEST_ROOT/repo/repo/nat/nat11
    
        tree $MOVE_TEST_ROOT
    }
    
    test_rsync_merge_folders(){
        __doc__="
        source ~/misc/tests/bash/test_rsync.sh
        "
        reset_rsync_test_local_move
    
        # Does not work
        #mv $MOVE_TEST_ROOT/repo/repo/* $MOVE_TEST_ROOT/repo
        rsync -avrRP $MOVE_TEST_ROOT/repo/./repo $MOVE_TEST_ROOT
    
        tree $MOVE_TEST_ROOT
    
        # Check the content of prime7 to see if it was overwritten or not
        # Ans: the data is not overwritten, only disjoint files are merged in
        cat $MOVE_TEST_ROOT/repo/primes/prime07
    
        # Remove the botched repo
        rm -rf $MOVE_TEST_ROOT/repo/repo
    
        # Note that the broken (nat11) link is overwritten
        tree $MOVE_TEST_ROOT
    
    }

The above script will create a test directory with a "correct" and a "botched" repo. Effectively we are supposed to have a repo with folders for natural, prime, and perfect numbers, but something went wrong and some of the data exists in the correct location, but we accidently made a subfolder repo/repo that contains part of the data. The goal is we want to merge everything from ./repo/repo into ./repo

The initial directory structure looks like this:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime11
    ├── perfect
    ├── primes
    │   ├── prime02
    │   ├── prime05
    │   ├── prime07
    │   └── prime13
    └── repo
        ├── nat
        │   ├── nat03 -> ../primes/prime03
        │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
        │   ├── nat08
        │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
        ├── perfect
        │   ├── perfect006
        │   ├── perfect028
        │   └── perfect496
        └── primes
            ├── prime03
            ├── prime07
            └── prime11

Note I threw in some relative and absolute symlinks to test how it works with those.

After executing:

rsync -avrRP $MOVE_TEST_ROOT/repo/./repo $MOVE_TEST_ROOT

We get:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat03 -> ../primes/prime03
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
    │   ├── nat08
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
    ├── perfect
    │   ├── perfect006
    │   ├── perfect028
    │   └── perfect496
    ├── primes
    │   ├── prime02
    │   ├── prime03
    │   ├── prime05
    │   ├── prime07
    │   ├── prime11
    │   └── prime13
    └── repo
        ├── nat
        │   ├── nat03 -> ../primes/prime03
        │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
        │   ├── nat08
        │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
        ├── perfect
        │   ├── perfect006
        │   ├── perfect028
        │   └── perfect496
        └── primes
            ├── prime03
            ├── prime07
            └── prime11

where everything is almost correctly moved. The only issue is that nat11 in the "correct" repo was a broken symlink, so that was overwritten with data from the "botched" subrepo. Other files are not overwritten, only disjoint data is merged.

Removing the botched subdir gives us:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat03 -> ../primes/prime03
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
    │   ├── nat08
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
    ├── perfect
    │   ├── perfect006
    │   ├── perfect028
    │   └── perfect496
    └── primes
        ├── prime02
        ├── prime03
        ├── prime05
        ├── prime07
        ├── prime11
        └── prime13

So, the rsync method mostly works, just be careful because any symlinks that are not resolved might be overwritten.

Answered By: Erotemic
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.