worker10 crashed triggering systemd-services alert and host-up alert size:M
It is up and running again. I've been moving the crash dump to
/var/crash-bak/2023-03-13-00:16 on that host.
Note that we've recently seen a crash on worker11 (#125207) and worker13 (#125210) as well.
- AC1: We know why certain workers have started to crash in similar ways recently
- Reach out for help on public channels or people from kernel testing squad
- Search relevant for already reported upstream bugs
#6 Updated by MDoucha 3 months ago
I've reported some new workqueue lockup warnings on SLE-15SP4 kernels in bsc#1201188 but this doesn't seem to be the same issue. All 3 worker however ran Btrfs rebalance shortly (~20 minutes) before the crash. Can you trigger the crash by running Btrfs rebalance manually? And if so, can you trigger it again when you boot the worker without the igb driver module?
#7 Updated by mkittler 3 months ago
All 3 worker however ran Btrfs rebalance shortly (~20 minutes) before the crash.
I've of course noticed that as well but I'm not sure whether it is related.
Can you trigger the crash by running Btrfs rebalance manually?
Good idea, I'll try that.
EDIT: I've just invoked
sudo systemctl start btrfs-balance.service on all three workers (worker10, 11 and 13) to increase the chances to reproduce the issue. So far it doesn't lead to any crashes and the re-balancing has already ended on all hosts. Maybe this isn't an ideal way to reproduce the problem because there was likely not much to re-balance; the re-balancing didn't take very long.
I've just triggered one more re-balancing on all three hosts but it also just exited very quickly with no crashes yet.