QA » openQA Project » openQA Infrastructure

Observation¶

From https://suse.slack.com/archives/C02D16TCP99/p1695966428720309

(Timo Jyrinki) Did we get enough updates published on Wednesday, or should someone be giving a little nudge to openQA to get today's tests running, as there has been nothing running for now?
(Oliver Kurz) The reason is https://suse.slack.com/archives/C02B5UVEC94/p1695921343628859 . Do you have an idea how to work around that? One could run qem-bot from another environment but I don't plan us to do that.
(Lars Vogdt) :alert: Our Container registry registry.opensuse.org is currently down, after a reboot to get to the current kernel, it got stuck in xfs ... :alert:
:eyes_right: at the moment we are trying to run xfs_repair, but get a a very low disc speed from the underlying storage device. So please expect the system to be unavailable for the next hours. :eyes:
Posted in engineering | Yesterday at 19:15 | View message
(Timo Jyrinki) Oh, ok, thank you for the information. I do not know how to operate the qem-bot, but maybe someone else knows? Especially if there's something urgent in the update queue @Heiko Rommel
(Oliver Kurz) it's not too hard to call locally, see https://github.com/openSUSE/qem-bot/#usage . However it seems registry.opensuse.org is up by now again. So I retriggered the call for aggregates: https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1865825, currently running

Problem¶

gitlab CI jobs rely on a container registry which was down