QA (public) » openQA Project (public) » openQA Infrastructure (public)

There seems to be some issue with openqaworker27.

Performing a power reset via IPMI using the command ssh -t jumpy@oqa-jumpy.dmz-prg2.suse.org "ipmitool -I lanplus -H openqaworker27.oqa-ipmi-ur -U oqadmin -P rumor-26Stamp power reset" was unsuccessful. After accessing the IPMI shell, I checked the power status, which showed "power off." I then executed power on to turn on the chassis power.

The machine is now reachable via ssh, but the openqa-worker related systemd unit files are not available on the system.

openqaworker27:~ # systemctl list-unit-files | grep openqa
var-lib-openqa.mount                    generated       -
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # systemctl list-units | grep openqa
  var-lib-openqa.mount                                                                      loaded active mounted   /var/lib/openqa
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # ls /var/lib/openqa/
cache  lost+found
openqaworker27:~ # ls -al /var/lib/openqa/
total 24
drwxr-xr-x 4 root root  4096 Feb  8  2024 .
drwxr-xr-x 1 root root   426 Sep  1  2023 ..
drwxr-xr-x 3 root root  4096 Feb  8  2024 cache
drwx------ 2 root root 16384 Aug  4  2023 lost+found
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # ls -al /var/lib/openqa/cache/
total 12
drwxr-xr-x 3 root root 4096 Feb  8  2024 .
drwxr-xr-x 4 root root 4096 Feb  8  2024 ..
drwxr-xr-x 2 root root 4096 Feb  8  2024 git
openqaworker27:~ # ls -al /var/lib/openqa/cache/git/
total 8
drwxr-xr-x 2 root root 4096 Feb  8  2024 .
drwxr-xr-x 3 root root 4096 Feb  8  2024 ..
openqaworker27:~ # 

Booted from an earlier snapshot hoping to find some useful/working installation of openQA-worker. But seems like openQA-worker wasn't installed at all?

openqaworker27:~ # 
openqaworker27:~ # systemctl list-unit-files | grep openqa
var-lib-openqa.mount                    generated       -
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # snapper list
 # | Type   | Pre # | Date                            | User | Used Space | Cleanup | Description           | Userdata     
---+--------+-------+---------------------------------+------+------------+---------+-----------------------+--------------
0  | single |       |                                 | root |            |         | current               |              
1+ | single |       | Fri 04 Aug 2023 03:28:22 PM UTC | root |  95.37 MiB |         | first root filesystem |              
2  | single |       | Fri 04 Aug 2023 03:36:42 PM UTC | root |   4.97 MiB | number  | after installation    | important=yes
3  | pre    |       | Sat 16 Dec 2023 06:11:32 PM UTC | root |   2.31 MiB | number  | zypp(zypper)          | important=no 
4- | post   |     3 | Sat 16 Dec 2023 06:11:34 PM UTC | root |   2.75 MiB | number  |                       | important=no 
5  | pre    |       | Thu 08 Feb 2024 02:09:40 PM UTC | root |   1.41 MiB | number  | zypp(zypper)          | important=no 
6  | post   |     5 | Thu 08 Feb 2024 02:09:41 PM UTC | root |   2.10 MiB | number  |                       | important=no 
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # 
openqaworker27:~ # cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.14.21-150500.53-default root=UUID=e144bf52-766d-4e06-8823-1ae071e468c7 rootflags=subvol=/@/.snapshots/4/snapshot linux console=tty console=ttyS1,115200 preempt=full mitigations=auto quiet security=apparmor
openqaworker27:~ # 

@okurz @livdywan

Should I go ahead and upgrade this one as well to Leap 15.6?
We also need to install, configure and register openQA-worker on this machine.
How many instances of workers were previously running on openqaworker27?

Please check earlier tickets what the plan was regarding w27. Maybe the machine was configured as bare-metal worker?

@nicksinger confirmed in his comment https://progress.opensuse.org/issues/154156#note-26 that worker27 doesn't have openQA setup.

gpathak wrote in #note-8:

@nicksinger confirmed in his comment https://progress.opensuse.org/issues/154156#note-26 that worker27 doesn't have openQA setup.

@okurz
Should we continue with the installation and setup of the openQA workers in the same ticket, or should we create a new one since this ticket was only intended to assess the current state of the openqaworker27 machine?

gpathak wrote in #note-9:

gpathak wrote in #note-8:

@nicksinger confirmed in his comment https://progress.opensuse.org/issues/154156#note-26 that worker27 doesn't have openQA setup.

@okurz
Should we continue with the installation and setup of the openQA workers in the same ticket, or should we create a new one since this ticket was only intended to assess the current state of the openqaworker27 machine?

do whatever is easiest for you but the AC would cover a re-installation as well so you it is up to you how to proceed.

I found #132134-59 where dheidler stated "Commented out gre entries for ow27 and ow28 on all workers in /etc/wicked/scripts/gre_tunnel_preup.sh as they are planned to be used as generalhw sut.". Can you please check with dheidler what happened after that? In the end we want to have at least some bare-metal test machines on o3

okurz wrote in #note-14:

As noted in #132647 it was originally planned to use w27+w28 as bare-metal test machines but then we settled with only amd-zen2-gpu-sut1.oqa.opensuse.org https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=16386 which by now is still online but unused for o3 openQA tests, see https://openqa.opensuse.org/admin/workers/1279

So please setup w27+w28 as generic qemu openQA workers

w28 is configured to run riscv tests.
Do we want to configure it as generic qemu workers?

openqaworker27 is now up and running tests in o3.

• https://openqa.opensuse.org/admin/workers/1371
• https://openqa.opensuse.org/admin/workers/1376
• https://openqa.opensuse.org/admin/workers/1377
• https://openqa.opensuse.org/admin/workers/1378
• https://openqa.opensuse.org/admin/workers/1387
• https://openqa.opensuse.org/admin/workers/1389
• https://openqa.opensuse.org/admin/workers/1393
• https://openqa.opensuse.org/admin/workers/1394
• https://openqa.opensuse.org/admin/workers/1399

gpathak wrote in #note-16:

okurz wrote in #note-14:

As noted in #132647 it was originally planned to use w27+w28 as bare-metal test machines but then we settled with only amd-zen2-gpu-sut1.oqa.opensuse.org https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=16386 which by now is still online but unused for o3 openQA tests, see https://openqa.opensuse.org/admin/workers/1279

So please setup w27+w28 as generic qemu openQA workers

w28 is configured to run riscv tests.
Do we want to configure it as generic qemu workers?

No, that's fine. Keep it as is

Removed multi-machine setup and related configuration from openqaworker27.

openqaworker27 and IPMI is unreachable, created a SD request to powercycle thhe machine.
Let's block on: https://sd.suse.com/servicedesk/customer/portal/1/SD-173293