Emergency mode after resizing logical volumes on Ubuntu server

I run the latest version of ubuntu server on a personal computer at home that I use as a server.
I have two logical volumes on it. vg0 and vg1. Recently, I noticed that my partition /dev/vg0/lv-var mounted at /var was too small. I backed up everything in /var and /home and proceeded to reduce the /dev/vg0/lv-home partition and increase the /dev/vg0/lv-var partition.
The commands I used looked like this:

umount /home
lvreduce --size 325G /dev/vg0/lv-home
lvextend -L +100.22G /dev/vg0/lv-var
xfs_growfs /dev/mapper/vg0-lv--var

Everthing seemed to have worked. Notice that I did not umount /var though as it was used by many essential processes.

After a reboot, the system now boots into emergency mode I can’t even use a keyboard to troubleshoot because it has trouble handling usb connections and I get an error similar to unable to enumerate USB device

I feel that there might be an issue with the fact that I did not remount /home before rebooting.

Trying to mount home while in safe mode I get:

mount: /home: can't read superblock on /dev/mapper/vg0-lv--home.

When the system boots up I can see:

[FAILED] Failed to mount /home. (100.0% complete)

I tried xfs_repair but I can’t even mount the volume anywhere because I get can't read superblock on /dev/mapper/vg0-lv--home

I am considering deleting the logical volume, creating a new one and mounting it to home. I have backups of my data, but how can I do this and allow the system to know to mount the newly created /home on startup?

Asked By: lucienfradet

||

I backed up everything in /var and /home

That’s great! You’ll need that backup of /home now.

You used lvreduce without --resizefs and without first shrinking the filesystem inside it. As a result, you have cut off part of your /home filesystem, and it’s now damaged and causing you to drop into emergency mode.

If your /home is an XFS filesystem, it is currently impossible to shrink it non-destructively anyway. Making a backup, then shrinking the partition/logical volume, then re-mkfsing the shrunk partition/LV and restoring the backup is for now the only available (production-quality) way to shrink an XFS filesystem.

So, temporarily comment out the /home entry in your /etc/fstab in emergency mode to let the system come up more or less normally, boot to get out of emergency mode, then re-mkfs the /dev/vg0/lv-home, uncomment the /etc/fstab entry for /home, mount it, and restore the contents of your /home from backup.

You should think about it in layers:

  • the physical disk is fine
  • the partition used as the LVM physical volume is fine
  • the LVM logical volume layer was properly shrunk, so it is also fine
  • the filesystem within the logical volume was not shrunk, so it’s had its tail end cut off.

If the filesystem type on /home was XFS, you’ll just need to run sudo mkfs.xfs /dev/vg0/lv-home and the filesystem will be good to mount again (although empty). After that, it’ll be just a matter of restoring the contents from backup.

Answered By: telcoM
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.