action #69523
Updated by okurz over 3 years ago
## Observation After https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/314 enabled automatic reboot of osd itself the VM did not come up from reboot on 2020-08-02 . okurz reported https://infra.nue.suse.com/SelfService/Display.html?id=175461 which bmwiedemann could resolve early 2020-08-03 . okurz logged in as root with the SSH key from "backup-vm" as normal user login was not working. Many services were not running. The mounted partition were not in order: ``` # df -h Filesystem Size Used Avail Use% Mounted on /dev/vda1 9.6G 5.8G 3.4G 63% / /dev/vdc 5.0T 4.5T 605G 89% /srv /dev/vdd 5.0T 3.4T 1.7T 68% /assets ``` missing /home and /results and also /srv should not be 5TB. ## Problem During the course of the last two years likely the former volume "vdb" was removed during runtime of the VM at the time when coolo and EngInfra moved assets and results to two new, separated volumes. However as /etc/fstab relies on the order of detected devices after former vdb vanished on the next bootup the existing partitions are now identified differently and were assigned to incorrect mount points and also former "vde" was missing. ## Solution okurz fixed this with defining mount points in /etc/fstab using UUIDs: ``` # cat /etc/fstab devpts /dev/pts devpts mode=0620,gid=5 0 0 proc /proc proc defaults 0 0 sysfs /sys sysfs noauto 0 0 debugfs /sys/kernel/debug debugfs noauto 0 0 usbfs /proc/bus/usb usbfs noauto 0 0 tmpfs /run tmpfs noauto 0 0 # 7116dc72-ebc8-4b21-8847-b9f31dc95229 -> vda1 /dev/vda1 / ext3 defaults 1 1 # 2e55520d-2b90-4100-8892-025c5f4c9949 -> vda2 /dev/vda2 swap swap defaults 0 0 # 6c8044d6-5497-4db6-9714-89b76268121e -> vdb UUID=6c8044d6-5497-4db6-9714-89b76268121e /srv xfs defaults,logbsize=256k,noatime,nodiratime 1 2 /srv/PSQL10 /var/lib/pgsql none bind 0 0 # 3f003a69-c51e-4d79-8b83-906e7918bac4 -> vdc UUID=3f003a69-c51e-4d79-8b83-906e7918bac4 /assets xfs defaults,logbsize=256k,noatime,nodiratime 1 2 /assets /var/lib/openqa/share none bind 0 0 # 51d504aa-6f46-4b89-bcd9-b6cea7b8b755 -> vdd UUID=51d504aa-6f46-4b89-bcd9-b6cea7b8b755 /results xfs defaults,logbsize=256k,noatime,nodiratime 1 2 /results /var/lib/openqa none bind 0 0 /srv/homes.img /home ext4 defaults 1 1 ``` As an alternative labels could be used. However only "assets" is currently available with a label. Filesystem labels can be set with `tune2fs -L $LABEL /dev/vd$i` for ext2/3/4 or for xfs `xfs_admin -L $LABEL /dev/vd$i` which however needs unmounted volumes so something left for later. TDB