Migrating Docker Volumes

By default Docker stores its persistent data in /var/lib/docker folder on Linux systems. This folder is part of the OS disk and in a production (or even dev) setup you might want to store persistent data on a data disk. Many OS Disks come with limited size. On a server that I was working on the /var folder only had 8GB to it.

/dev/mapper/rootvg-varlv   7.8G  7.8G     0 100% /var

It got full fast and then we had a big problem. We had running containers on the machine. What to do now? First we had to attach a data disk to the machine. Once that is done follow the recipe below to migrate Docker’s storage to another folder while keeping everything intact! The instructions below are for Red Hat Enterprise Linux (RHEL) and so should work as-is on CentOS. On other systems you might need some tweaks to the commands.

The new location is /app which resides on a different disk in this writeup.

Step 1: Stop Docker daemon

#-> systemctl stop docker

Step 2: Copy /var/lib/docker to new location

Copy the whole of /var/lib/docker not just the volumes inside it or you will lose your containers etc.

#-> rsync -aP /var/lib/docker /app

Run ls /app/docker to verify everything is there.

#-> ls /app/docker
builder   containerd  image    overlay2  runtimes  tmp    volumes
buildkit  containers  network  plugins   swarm     trust

Step 3: Rename /var/lib/docker to /var/lib/docker.old

For safety, always retain the old directory.

#-> mv /var/lib/docker /var/lib/docker.old

To see the file size

#-> du -ch -d 1 /var/lib/docker.old
212K	/var/lib/docker.old/containerd
1.1G	/var/lib/docker.old/overlay2
72K	/var/lib/docker.old/buildkit
1.5M	/var/lib/docker.old/containers
20K	/var/lib/docker.old/builder
62M	/var/lib/docker.old/swarm
4.0K	/var/lib/docker.old/tmp
24K	/var/lib/docker.old/plugins
995M	/var/lib/docker.old/volumes
4.0K	/var/lib/docker.old/runtimes
4.0K	/var/lib/docker.old/trust
372K	/var/lib/docker.old/network
9.5M	/var/lib/docker.old/image
2.1G	/var/lib/docker.old
2.1G	total

WARNING: If you run mv /var/lib/docker /app/docker.old its going to hang for a while because /app is on another filesystem and so it has to really move the files to that folder – it can’t just rename the folder. So don’t do that. Run rsync again if you want to keep an archive of the data on /app.

Step 4: Add symlink from /var/lib/docker to /app/docker

Note the destination folder comes as the first argument to the ln command:

#-> ln -s /app/docker /var/lib/docker

The old containers have /var/lib/docker paths configured in them and so we need to setup this symlink so that we can restart the old containers and everything works.

Verify:

#-> ls -al /var/lib/docker
lrwxrwxrwx 1 root root 11 Sep 29 17:36 /var/lib/docker -> /app/docker

This is the error I got when trying to restart the container if I skipped this step and did not setup the symlink:

#-> docker start middleware-mysql
Error response from daemon: error evaluating symlinks from mount source "/var/lib/docker/volumes/middleware-mysql/_data": lstat /var/lib/docker/volumes: no such file or directory
Error: failed to start containers: middleware-mysql

Step 5: Edit /lib/systemd/system/docker.service and set the --data-root variable

The --data-root is explained below:

--data-root string                      Root directory of persistent Docker state (default "/var/lib/docker")

Edit following file:

#-> vi /lib/systemd/system/docker.service

and add entry for --data-root pointing it to /app/docker:

ExecStart=/usr/bin/dockerd --data-root /app/docker

Step 6: Reload configuration

#-> systemctl daemon-reload

Step 7: Restart Docker

#-> systemctl start docker

Run systemctl status docker to verify everything is good

Step 8: Restart containers

Finally just restart the containers:

#-> docker start foo
#-> docker start bar

That’s it! Everything is restored.

It was a nerve-racking experience for sure. Now I saw following when I did a df -h

#-> df -h
Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.9G     0  3.9G   0% /dev
tmpfs                      3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                      3.9G  378M  3.6G  10% /run
tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/rootvg-rootlv  7.8G  145M  7.2G   2% /
/dev/mapper/rootvg-usrlv   9.8G  2.1G  7.2G  23% /usr
/dev/sda1                  976M   83M  826M  10% /boot
/dev/sdc                   246G  2.1G  232G   1% /app
/dev/mapper/rootvg-homelv  976M  364M  546M  40% /home
/dev/mapper/rootvg-optlv   2.0G  1.6G  280M  85% /opt
/dev/mapper/rootvg-varlv   7.8G  7.5G     0 100% /var
/dev/mapper/rootvg-tmplv   2.0G  379M  1.5G  21% /tmp
/dev/sdb1                   16G  2.1G   13G  14% /mnt/resource
tmpfs                      797M     0  797M   0% /run/user/48081
overlay                    246G  2.1G  232G   1% /app/docker/overlay2/b65f1433823722f8bba50ff4461f8a8789a9f652f33532326c96bc6215af9d28/merged
shm                         64M     0   64M   0% /app/docker/containers/cf51fefa07b118eb252b382947650ad3fbd58a2dbeb2cd7c72e23ab1a0f9e862/mounts/shm
overlay                    246G  2.1G  232G   1% /app/docker/overlay2/ca0999cd7dee9d9b3a9911150ff4e6bc06d8c5552a9faa2158f3ecc35cce3a9f/merged
shm                         64M     0   64M   0% /app/docker/containers/1f1ff1dc1e4cf96decdd6ea57fe51a72c1edbfdad89771e396f217fc5d294383/mounts/shm
This entry was posted in Software. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s