Actions
action #65073
closedaarch64.o.o fails to delete snapshots in snapper-cleanup since 2020-03-20 repeatedly
Start date:
2020-03-31
Due date:
% Done:
0%
Estimated time:
Description
Observation¶
journalctl --since="2020-03-10" -u snapper-cleanup
Mar 19 03:45:11 openqa-aarch64 systemd-helper[6734]: running number cleanup for 'root'.
Mar 19 03:45:36 openqa-aarch64 systemd-helper[6734]: running timeline cleanup for 'root'.
Mar 19 03:45:36 openqa-aarch64 systemd-helper[6734]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 20 03:43:56 openqa-aarch64 systemd-helper[3375]: running number cleanup for 'root'.
Mar 20 03:43:56 openqa-aarch64 systemd-helper[3375]: Deleting snapshot failed.
Mar 20 03:43:56 openqa-aarch64 systemd-helper[3375]: number cleanup for 'root' failed.
Mar 20 03:43:56 openqa-aarch64 systemd-helper[3375]: running timeline cleanup for 'root'.
Mar 20 03:43:56 openqa-aarch64 systemd-helper[3375]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 21 03:43:59 openqa-aarch64 systemd-helper[3392]: running number cleanup for 'root'.
Mar 21 03:43:59 openqa-aarch64 systemd-helper[3392]: Deleting snapshot failed.
Mar 21 03:43:59 openqa-aarch64 systemd-helper[3392]: number cleanup for 'root' failed.
Mar 21 03:43:59 openqa-aarch64 systemd-helper[3392]: running timeline cleanup for 'root'.
Mar 21 03:43:59 openqa-aarch64 systemd-helper[3392]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 22 03:45:20 openqa-aarch64 systemd-helper[9743]: running number cleanup for 'root'.
Mar 22 03:45:20 openqa-aarch64 systemd-helper[9743]: Deleting snapshot failed.
Mar 22 03:45:20 openqa-aarch64 systemd-helper[9743]: number cleanup for 'root' failed.
Mar 22 03:45:20 openqa-aarch64 systemd-helper[9743]: running timeline cleanup for 'root'.
Mar 22 03:45:20 openqa-aarch64 systemd-helper[9743]: running empty-pre-post cleanup for 'root'.
Mar 23 03:45:30 openqa-aarch64 systemd-helper[6390]: running number cleanup for 'root'.
Mar 23 03:45:30 openqa-aarch64 systemd-helper[6390]: Deleting snapshot failed.
Mar 23 03:45:30 openqa-aarch64 systemd-helper[6390]: number cleanup for 'root' failed.
Mar 23 03:45:30 openqa-aarch64 systemd-helper[6390]: running timeline cleanup for 'root'.
Mar 23 03:45:30 openqa-aarch64 systemd-helper[6390]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 24 03:43:20 openqa-aarch64 systemd-helper[3370]: running number cleanup for 'root'.
Mar 24 03:43:20 openqa-aarch64 systemd-helper[3370]: Deleting snapshot failed.
Mar 24 03:43:20 openqa-aarch64 systemd-helper[3370]: number cleanup for 'root' failed.
Mar 24 03:43:20 openqa-aarch64 systemd-helper[3370]: running timeline cleanup for 'root'.
Mar 24 03:43:20 openqa-aarch64 systemd-helper[3370]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 25 03:43:24 openqa-aarch64 systemd-helper[3388]: running number cleanup for 'root'.
Mar 25 03:43:24 openqa-aarch64 systemd-helper[3388]: Deleting snapshot failed.
Mar 25 03:43:24 openqa-aarch64 systemd-helper[3388]: number cleanup for 'root' failed.
Mar 25 03:43:24 openqa-aarch64 systemd-helper[3388]: running timeline cleanup for 'root'.
Mar 25 03:43:24 openqa-aarch64 systemd-helper[3388]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 26 03:43:31 openqa-aarch64 systemd-helper[3371]: running number cleanup for 'root'.
Mar 26 03:43:31 openqa-aarch64 systemd-helper[3371]: Deleting snapshot failed.
Mar 26 03:43:31 openqa-aarch64 systemd-helper[3371]: number cleanup for 'root' failed.
Mar 26 03:43:31 openqa-aarch64 systemd-helper[3371]: running timeline cleanup for 'root'.
Mar 26 03:43:31 openqa-aarch64 systemd-helper[3371]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 26 20:03:59 openqa-aarch64 systemd-helper[6837]: running number cleanup for 'root'.
Mar 26 20:03:59 openqa-aarch64 systemd-helper[6837]: Deleting snapshot failed.
Mar 26 20:03:59 openqa-aarch64 systemd-helper[6837]: number cleanup for 'root' failed.
Mar 26 20:03:59 openqa-aarch64 systemd-helper[6837]: running timeline cleanup for 'root'.
Mar 26 20:03:59 openqa-aarch64 systemd-helper[6837]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 26 21:19:34 openqa-aarch64 systemd-helper[5000]: running number cleanup for 'root'.
Mar 26 21:19:34 openqa-aarch64 systemd-helper[5000]: Deleting snapshot failed.
Mar 26 21:19:34 openqa-aarch64 systemd-helper[5000]: number cleanup for 'root' failed.
Mar 26 21:19:34 openqa-aarch64 systemd-helper[5000]: running timeline cleanup for 'root'.
Mar 26 21:19:34 openqa-aarch64 systemd-helper[5000]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 27 19:40:10 openqa-aarch64 systemd-helper[4301]: running number cleanup for 'root'.
Mar 27 19:40:10 openqa-aarch64 systemd-helper[4301]: Deleting snapshot failed.
Mar 27 19:40:10 openqa-aarch64 systemd-helper[4301]: number cleanup for 'root' failed.
Mar 27 19:40:10 openqa-aarch64 systemd-helper[4301]: running timeline cleanup for 'root'.
Mar 27 19:40:10 openqa-aarch64 systemd-helper[4301]: running empty-pre-post cleanup for 'root'.
-- Reboot --
Mar 28 03:43:39 openqa-aarch64 systemd-helper[3376]: running number cleanup for 'root'.
Mar 28 03:43:39 openqa-aarch64 systemd-helper[3376]: Deleting snapshot failed.
Mar 28 03:43:39 openqa-aarch64 systemd-helper[3376]: number cleanup for 'root' failed.
Mar 28 03:43:39 openqa-aarch64 systemd-helper[3376]: running timeline cleanup for 'root'.
Mar 28 03:43:39 openqa-aarch64 systemd-helper[3376]: running empty-pre-post cleanup for 'root'.
shows that snapper-cleanup always fails since 2020-03-20
Updated by okurz over 4 years ago
- Related to action #62102: aarch64.o.o did not come up after nightly upgrade due to grub2-arm64-efi upgrade (boo#1162320) added
Updated by okurz over 4 years ago
- Status changed from New to In Progress
- Assignee set to okurz
seems my work in #62102 has caused this (or others debugging). The sub-subvolumes block the deletion:
# btrfs subvolume list /
ID 256 gen 912906 top level 5 path @
ID 258 gen 912913 top level 256 path @/var
ID 259 gen 912897 top level 256 path @/usr/local
ID 260 gen 912912 top level 256 path @/tmp
ID 261 gen 912897 top level 256 path @/srv
ID 262 gen 912897 top level 256 path @/root
ID 263 gen 456301 top level 256 path @/opt
ID 264 gen 912913 top level 256 path @/.snapshots
ID 917 gen 863529 top level 264 path @/.snapshots/569/snapshot
ID 918 gen 895319 top level 917 path @/.snapshots/569/snapshot/boot/grub2/arm64-efi
ID 921 gen 866968 top level 264 path @/.snapshots/571/snapshot
ID 922 gen 869316 top level 264 path @/.snapshots/572/snapshot
# btrfs subvolume delete @/.snapshots/569/snapshot/boot/grub2/arm64-efi
Delete subvolume (no-commit): '/mnt/@/.snapshots/569/snapshot/boot/grub2/arm64-efi'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
# mount -o subvolid=0 /dev/sda2 /mnt
# cd /mnt/
# btrfs subvolume delete @/.snapshots/568/snapshot/boot/grub2/arm64-efi
Delete subvolume (no-commit): '/mnt/@/.snapshots/568/snapshot/boot/grub2/arm64-efi'
# btrfs subvolume delete @/.snapshots/568/snapshot
Delete subvolume (no-commit): '/mnt/@/.snapshots/568/snapshot'
# btrfs subvolume delete --commit-each @/.snapshots/569/snapshot/boot/grub2/arm64-efi
Delete subvolume (commit): '/mnt/@/.snapshots/569/snapshot/boot/grub2/arm64-efi'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
# btrfs subvolume delete @/.snapshots/569/snapshot/boot/grub2/arm64-efi
Delete subvolume (no-commit): '/mnt/@/.snapshots/569/snapshot/boot/grub2/arm64-efi'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
# transactional-update dup && reboot
Updated by okurz over 4 years ago
I managed to delete snapshots 568 but failed to do the same with 569
Updated by favogt over 4 years ago
- Status changed from In Progress to Feedback
Deleted manually and triggered a cleanup:
openqa-aarch64:~ # btrfs subvol del /.snapshots/569/snapshot/boot/grub2/arm64-efi
Delete subvolume (no-commit): '/.snapshots/569/snapshot/boot/grub2/arm64-efi'
ERROR: Could not destroy subvolume/snapshot: Read-only file system
openqa-aarch64:~ # btrfs subvol del /.snapshots/569/snapshot
Delete subvolume (no-commit): '/.snapshots/569/snapshot'
ERROR: Could not destroy subvolume/snapshot: Directory not empty
openqa-aarch64:~ # btrfs prop set /.snapshots/569/snapshot ro false
openqa-aarch64:~ # btrfs subvol del /.snapshots/569/snapshot/boot/grub2/arm64-efi
Delete subvolume (no-commit): '/.snapshots/569/snapshot/boot/grub2/arm64-efi'
openqa-aarch64:~ # btrfs prop set /.snapshots/569/snapshot ro true
openqa-aarch64:~ # systemctl start snapper-cleanup
openqa-aarch64:~ # systemctl status snapper-cleanup
● snapper-cleanup.service - Daily Cleanup of Snapper Snapshots
Loaded: loaded (/usr/lib/systemd/system/snapper-cleanup.service; static; vendor preset: disabled)
Active: active (running) since Mon 2020-04-06 14:48:38 CEST; 946ms ago
Docs: man:snapper(8)
man:snapper-configs(5)
Main PID: 15726 (systemd-helper)
Tasks: 1
CGroup: /system.slice/snapper-cleanup.service
└─15726 /usr/lib/snapper/systemd-helper --cleanup
Apr 06 14:48:38 openqa-aarch64 systemd-helper[15726]: running number cleanup for 'root'.
openqa-aarch64:~ # systemctl status snapper-cleanup
● snapper-cleanup.service - Daily Cleanup of Snapper Snapshots
Loaded: loaded (/usr/lib/systemd/system/snapper-cleanup.service; static; vendor preset: disabled)
Active: inactive (dead) since Mon 2020-04-06 14:49:48 CEST; 39s ago
Docs: man:snapper(8)
man:snapper-configs(5)
Process: 15726 ExecStart=/usr/lib/snapper/systemd-helper --cleanup (code=exited, status=0/SUCCESS)
Main PID: 15726 (code=exited, status=0/SUCCESS)
Apr 06 14:48:38 openqa-aarch64 systemd-helper[15726]: running number cleanup for 'root'.
Apr 06 14:49:48 openqa-aarch64 systemd-helper[15726]: running timeline cleanup for 'root'.
Apr 06 14:49:48 openqa-aarch64 systemd-helper[15726]: running empty-pre-post cleanup for 'root'.
Updated by okurz over 4 years ago
asked fvogt for help as he was doing experiments on that machine previously as well.
Updated by okurz over 4 years ago
- Status changed from Feedback to Resolved
Thank you. The btrfs prop set /.snapshots/569/snapshot ro false was new to me and the missing part, interesting :)
Actions