ZFS bpool is almost full; how can I free up space so I can keep updating my system without errors?
I am running Ubuntu with ZFS on root, and my bpool seems to be almost full.
Output of zpool list bpool:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 1.69G 191M - - 60% 90% 1.00x ONLINE -
Output of df -h /boot:
Filesystem Size Used Avail Use% Mounted on
bpool/BOOT/ubuntu_s2c09x 367M 303M 64M 83% /boot
This is causing problems with completing the installation of kernel updates, e.g. running sudo apt upgrade:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Requesting to save current system state
ERROR couldn't save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool "bpool" is 10%.
Please remove some states manually to free up space.
Setting up linux-firmware (1.197.5) ...
update-initramfs: Generating /boot/initrd.img-5.11.0-41-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-41-generic with 1.
dpkg: error processing package linux-firmware (--configure):
installed linux-firmware package post-installation script subprocess returned error exit status 1
Setting up linux-image-5.11.0-34-generic (5.11.0-34.36) ...
Processing triggers for linux-image-5.11.0-34-generic (5.11.0-34.36) ...
/etc/kernel/postinst.d/dkms:
* dkms: running auto installation service for kernel 5.11.0-34-generic
Error! Your kernel headers for kernel 5.11.0-34-generic cannot be found.
Please install the linux-headers-5.11.0-34-generic package,
or use the --kernelsourcedir option to tell DKMS where it's located
...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.11.0-34-generic
I: The initramfs will attempt to resume from /dev/nvme0n1p2
I: (UUID=abdb40da-2d0b-4a45-b024-1c7245ea27ea)
I: Set the RESUME variable to override this.
Error 24 : Write error : cannot write compressed block
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.11.0-34-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.11.0-34-generic (--configure):
installed linux-image-5.11.0-34-generic package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
linux-firmware
linux-image-5.11.0-34-generic
ZSys is adding automatic system snapshot to GRUB menu
I’ve tried destroying some bpool snapshots, but zfs list -t snapshot | grep bpool doesn’t list any over 100K. I have also tried removing some kernels, see output of dpkg -l linux-{image,headers}-"[0-9]*":
un linux-headers-3.0 <none> <none> (no description available)
un linux-headers-5.10.0-1045-oem <none> <none> (no description available)
un linux-headers-5.11.0-34-generic <none> <none> (no description available)
un linux-headers-5.11.0-35-generic <none> <none> (no description available)
un linux-headers-5.11.0-36-generic <none> <none> (no description available)
un linux-headers-5.11.0-37-generic <none> <none> (no description available)
un linux-headers-5.11.0-38-generic <none> <none> (no description available)
ii linux-headers-5.11.0-40 5.11.0-40.44 all Header files related to Linux kernel version 5.11.0
ii linux-headers-5.11.0-40-generic 5.11.0-40.44 amd64 Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
ii linux-headers-5.11.0-41 5.11.0-41.45 all Header files related to Linux kernel version 5.11.0
ii linux-headers-5.11.0-41-generic 5.11.0-41.45 amd64 Linux kernel headers for version 5.11.0 on 64 bit x86 SMP
un linux-headers-686-pae <none> <none> (no description available)
ic linux-image-5.10.0-1045-oem 5.10.0-1045.47 amd64 Signed kernel image oem
un linux-image-5.11.0-27-generic <none> <none> (no description available)
iF linux-image-5.11.0-34-generic 5.11.0-34.36 amd64 Signed kernel image generic
rc linux-image-5.11.0-35-generic 5.11.0-35.37 amd64 Signed kernel image generic
rc linux-image-5.11.0-36-generic 5.11.0-36.40 amd64 Signed kernel image generic
rc linux-image-5.11.0-37-generic 5.11.0-37.41 amd64 Signed kernel image generic
rc linux-image-5.11.0-38-generic 5.11.0-38.42 amd64 Signed kernel image generic
ii linux-image-5.11.0-40-generic 5.11.0-40.44 amd64 Signed kernel image generic
ii linux-image-5.11.0-41-generic 5.11.0-41.45 amd64 Signed kernel image generic
The oldest currently installed one seems to be linux-image-5.11.0-34-generic, however sudo apt purge linux-image-5.11.0-34-generic suggests that it wants to also install linux-image-unsigned-5.11.0-34-generic, taking up more space.
What else can I try? Is it conceivably an option to resize/grow my bpool? I am pretty new to using ZFS.
Edit: I did some more investigating. zfs list bpool/BOOT/ubuntu_s2c09x -o space -r reports
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
bpool/BOOT/ubuntu_s2c09x 229M 1.51G 1.22G 303M 0B 0B
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-19-1344 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-21-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-22-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-23-1340 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-24-1335 - 56K - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-25-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-26-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-11-26-1343 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-27-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-28-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-29-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-11-30-1340 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-01-1340 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-02-1340 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-03-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-03-1344 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-04-1337 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-05-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-06-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-07-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-08-1337 - 80K - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-09-0645 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_a2u0ly - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-10-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-10-1341 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-11-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-12-1335 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_omna1o - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_5q7106 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-13-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_fwalez - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-14-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_dsm4th - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-15-1335 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_4t769s - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-16-1336 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_z8aa09 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-17-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_weekly-2021-12-17-1344 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_dqsnib - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_7w0l6s - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_323q08 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@autozsys_gqbsa5 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-18-1338 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-19-1339 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0217 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0317 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0417 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0517 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0617 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0717 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0817 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-0917 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1017 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1117 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1217 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1317 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_daily-2021-12-21-1337 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1417 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1517 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1617 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1717 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1817 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-1917 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2017 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2117 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2217 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-21-2317 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0017 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0100 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0115 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_hourly-2021-12-22-0117 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0130 - 0B - - - -
bpool/BOOT/ubuntu_s2c09x@zfs-auto-snap_frequent-2021-12-22-0145 - 0B - - - -
I don’t understand how it can claim that 1.22 GB are used by snapshots, but none of the listed bpool snapshots are using any space?
Uninstalling old and unused kernels and associated files is the correct action to take; to avoid the suggestion to install an unsigned kernel, remove packages that depend on linux-image-5.11.0-34-generic via:
sudo apt purge linux*5.11.0-34*
The second step required to free up space is to remove old snapshots referencing the files that were removed; list the snapshots via:
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT
You can see in the REFER column how much data is referenced. Identical numbers often coincide with minimal USED values; the snapshots are effectively copies of previous snapshots.
Manually remove the five oldest snapshots via:
zfs list -H -r -t snapshot -o name,used,referenced,creation bpool/BOOT | head -n 5 | cut -c 35-40 | xargs -n 1 sudo zsysctl state remove --system
Keep removing snapshots until you see space being freed up.
(Source of the last one-liner and more info: https://github.com/ubuntu/zsys/issues/155)
Notes:
-
Growing a ZFS pool is possible if there is unallocated space, but Ubuntu allocates the remaining space to
rpool. Shrinking a ZFS pool in-place is apparently not possible. -
I see auto-snapshotting is turned on for
bpool, but unless you expect any files to change outside of APT, ZSys snapshotting should be sufficient and would reduce the number of snapshots you have to consider. -
To avoid this problem in the future, stay on top of kernel upgrades and remove old ones regularly. Also consider reducing the ZSys garbage cleaning defaults in
/etc/zsys.conf. For an example, see: https://github.com/ubuntu/zsys/issues/155#issuecomment-758902487
This will remove the oldest 5 kernels from zfs. It will not remove the most recent 5 if there are 5 or fewer kernels installed.
# Run "zfs list" to list all the installed kernels.
# Tail removes the header line
# head -n -5 preserves the last 5 in the list, listing all others
# head -5 removes only the first 5 (if at least that many exist)
zfs list -r -t snapshot -o name,used,referenced,creation bpool/BOOT | tail -n +2 | head -n -5 | head -5 | cut -c 35-40 | xargs -n 1 /usr/bin/sudo zsysctl state remove --system
# List the amount of space remaining on /boot
df -h /boot