QA (public) » openQA Project (public) » openQA Infrastructure (public)

ybonatakis wrote:

I tried to access AWS following https://progress.opensuse.org/projects/openqav3/wiki#Fallback-deployment-on-AWS but I couldnt.

That's nonsense. Those are only instructions how one could setup an alternative infrastructure if the original one is not usable for long, not related to needing to recover the original instance

That's what I wrote in the ticket

I assume part of the root filesystem, potentially more, was removed by accidental user action. If you can confirm that the system is inoperable please recover snapshots of the filesystem images attached to openqa.suse.de to the most recent state before 2025-04-19 05:00 UTC

No response since some hours. Since I was involved in multiple urgent mitigations here I would prefer if someone else picks this up and cleans up the mess :)

I disabled osd-deployment and salt-states-openqa pipelines, so when the VM is back, we can check everything before running deployment and salt.

edit: and also salt-pillars-openqa

Also disabled backup for now

Also disabled fetch_openqa_bugs

Also disabled the other backup on backup.qe.prg2.suse.org

Silent alerts as for now:
qesapworker-prg6 hostup
worker33
schort-server
worker29
worker31
worker-arm1
worker-arm2
tumblesle
backup
netboot.qe.prg2.suse.org
worker30
storage
backup-vm
worker34
diesel
petrol
osiris-1
backup-qam
worker35
s390zl12
monitor
worker36
sapworker1
grenache-1
qamaster
unreal6
baremetal-support
baremetal-support-prg2
jenkins
netboot
qesapworker-prg7
worker32
openqaw5-xen
mania
ada

I disabled salt-minion.service on both backup hosts now, as apparently at least on backup.qe.prg2.suse.org it was still running somehow and overwrote the rsnapshot.conf

this is now being worked on by Ignacio Torres and I asked him in https://suse.slack.com/archives/C029APBKLGK/p1745311147032479 to continue in a group chat

For a bit more context from conversation with Aziz Rozyev and Ignacio Torres from IT:
There are only 2 daily snapshots recorded, so the most recent we have for the root disk are:

                  weekly.2025-04-13_0015                 237.0GB     1%    4%
                  weekly.2025-04-20_0015                      0B     0%    0%
                  daily.2025-04-21_0010                       0B     0%    0%

so we went for the weekly.2025-04-13_0015. All 5 storage volumes were recovered. Ignacio first booted the system as we requested with systemd.unit=emergency.target. I provided the root password and Ignacio could login and mask+disable openqa-scheduler and openqa-webui. After that the VM was rebooted and we could login over ssh and continue. I have now enabled again osd-deployment and triggered it, also scripts-ci, salt-states-openqa and salt-pillars-openqa. tinita has bumped the auto-increment id for openQA jobs to prevent conflicts based on maximum recorded in http://dashboard.qam.suse.de/. Then new jobs have been triggered and show the new id 1740000+. Enabled openqa-webui and openqa-scheduler now. All looks good so far.

backup-vm.qe.nue2.suse.org

backup  root@localhost:/etc/    openqa.suse.de/ ssh_args=-p2222
backup_exec     ssh -p 2222 root@localhost "cd /tmp; sudo -u postgres ionice -c3 nice -n19 pg_dump -Fc openqa -f /var/lib/openqa/SQL-DUMPS/$(date -I).dump"
backup  root@localhost:/var/lib/openqa/SQL-DUMPS/       openqa.suse.de/ ssh_args=-p2222
backup  root@localhost:/var/log/zypp/   openqa.suse.de/ ssh_args=-p2222```

Only step missing from rollbacks are the silent alerts

https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs?statuses=WAITING_FOR_RESOURCE is showing 300+ jobs meaning that there is a longer backlog. You can cancel some "schedule incident" jobs which should clean up the queue a bit.
I wrote in #eng-testing https://suse.slack.com/archives/C02CANHLANP/p1745331064968439?thread_ts=1745043127.805829&cid=C02CANHLANP

Greetings from the past! https://openqa.suse.de is back in operation based on a state from 2025-04-13 which was the most recent consistent snapshot state that the backup system has. We are carefully monitoring the system and retriggering builds and jobs as applicable. Feel welcome to also trigger according products yourself as needed.

Setting due date based on mean cycle time of SUSE QE Tools

I removed the silences. and I will put this ticket in feedback as we still monitoring the server in one or the other way.

• AC1: openqa.suse.de is accessible in a web browser
• AC2: NFS mount and all related filesystems are back and working as previously
• AC3: All workers are connected and accessible via salt
• AC4: The web UI looks sensible

I consider all the ACs done. closing this one and following up to the subtasks that other colleagues have created.

https://gitlab.suse.de/openqa/osd-deployment/-/pipeline_schedules is still inactive

We fixed that. Probably I messed up.