QA » openQA Project » openQA Infrastructure

The sensor issues are a red herring, the values probably only show up if the system is fully booted.

Only the fans FAN1, FAN3 and FANA have "health status" in the BMC and it is critical. Maybe just wrong sensor reading (as most sensor data is in fact missing) but it could also be the culprit. That disabling CPUs helped for a while would be in accordance with and overheating system due to fan issues.

Note that I tried powering on the system again. However, the system got stuck even before reaching the bootloader. Then I tried it again and it could reach the bootloader and I could select the UEFI firmware menu from there. While being in the firmeware menu, the sensor readings on the BMC interface were unchanged. Unfortunately the UEFI menu itself doesn't show any interesting information (which would e.g. confirm that fans are broken). I only found a few entries in the event log which I've attached.

I've just created an infra ticket: https://sd.suse.com/servicedesk/customer/portal/1/SD-96734

The only thing I can think of we could do from our side is a firmware update. However, if it is really a fan that might not be a good idea. In general, if it is really a fan we should likely refrain from doing too much.

Looks like we have to contact delta computers.

Got an answer from DELTA. Apparently the CPU was broken and they plan to swap it with a different one. According to the mail from DELTA they plan to ship it back to us (Frankenstraße) beginning next week (CW40).

Sehr geehrte Damen und Herren,
ihr reparierter Rechner wurde heute per Spedition versendet. Im Anhang erhalten Sie Ihren elektronischen Lieferschein mit der Nummer 21660.
Sobald Ihre Sendung beim Spediteur verladen wurde, sehen Sie den Zustellstatus Ihrer Lieferung. Dieser ist in der Regel ab 8.00 Uhr, einen Tag nach Erhalt dieser Email, für Sie ersichtlich.
Die Lieferzeit für nationale Sendungen beträgt in der Regel 1-2 Werktage.

Mit dem folgenden Link können Sie Ihre Sendung direkt verfolgen:

https://www.ids-logistik.de/de/sendungsverfolgung?tracking=90461_2009043476370005

openqaworker20 is back in town! (Frankencampus, new qe cold storage area, Riegel 2 (the middle one, if you come from the kitchen elevator, the shared space to the right)

@nicksinger have you organized transport to Maxtorhof? Or any other related tasks? As https://www.ids-logistik.de/de/sendungsverfolgung?tracking=90461_2009043476370005 states "Zugestellt" this shouldn't stay on "Blocked" anymore.

Indeed this should not be blocked any longer. I've asked Oliver Fecher in a private slack message if he can help us to bring the machine back into maxtorhof.

Robert volunteered to ask in person, since he's in the office today

I forgot to bump the due date. Let's see where we get by the end of the week

SD-Ticket regarding setup at SRV1:
https://sd.suse.com/servicedesk/customer/portal/1/SD-102481

Nick and me will try to meet Gerhard or Max and solve that in place at Maxtorhof.

IPMI can now be accessed with the credentials noted in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L60

This ticket has no AC. I suggest we estimate it again and fill in the gaps.

I was so free to add acceptance criteria although this should be quite self-explaining.

I was so free to add acceptance criteria although this should be quite self-explaining.

Apparently openqaworker20 was moved to the new security zone already, so I had to request access to qe-jumpy. After that was granted I could set the machine up again. So far a simple (xfce-live) test passed and a local MM test scenario (firewalld-policy-*) also works.

Here are my notes of what I did, basically copy-pasting commands after I ran them. Apparently using code blocks in ordered lists doesn't quite work right here, so feel free to count yourself.

1. Enabled the tftp + uefi parts in /etc/dnsmasq.d/pxeboot.conf again. I had to make the efi file unconditional, the tag did somehow not match and it defaulted to grub for ppc64 (visible in dnsmasq's log).
2. Booted the server into IPv4 PXE mode on the right interface.
3. Completed the installation over ipmitool SoL, enabled SSH service + opened ports.
4. Came up after a reboot. Initial setup:

zypper ar -p 95 'http://download.opensuse.org/repositories/devel:openQA/$releasever' devel_openQA
zypper ar -p 90 'http://download.opensuse.org/repositories/devel:openQA:Leap:15.4/15.4' devel_openQA_Leap
zypper -v ref
zypper in bash-completion nano # yes :P
# relogin to activate bash completion
zypper in os-autoinst-distri-opensuse-deps
zypper in os-autoinst-openvswitch
nano /etc/openqa/client.conf # fill with data
nano /etc/openqa/workes.ini # copied from ow19, WORKER_CLASS adjusted
nano /etc/fstab # add /var/lib/openqa/share mount, copied from ow19
zypper in nfs-client
systemctl start var-lib-openqa-share.automount
zypper in openQA-worker # Should've done that earlier, thought it was included already:
# warning: /etc/openqa/client.conf created as /etc/openqa/client.conf.rpmnew
# Detected autofs mount point /var/lib/openqa/share during canonicalization of /var/lib/openqa/share.
# Skipping /var/lib/openqa/share
systemctl enable openqa-worker-auto-restart@{1..20}
mv /etc/openqa/workes.ini /etc/openqa/workers.ini # To fix the earlier typo...
systemctl enable --now openqa-worker-cacheservice openqa-worker-cacheservice-minion
systemctl start openqa-worker-auto-restart@1
systemctl disable --now salt-minion.service # not used yet
# Clone a test job (xfce-live): openqa-clone-job --within-instance https://openqa.opensuse.org 2927811 _GROUP=38 WORKER_CLASS=openqaworker20
zypper in --oldpackage http://download.opensuse.org/update/leap/15.3/sle/noarch/qemu-ovmf-x86_64-202008-150300.10.14.1.noarch.rpm
zypper al -m "poo#116914" qemu-ovmf-x86_64
zypper in ffmpeg-4 # Referenced in workers.ini
firewall-cmd --permanent --new-service isotovideo
for i in {1..50}; do firewall-cmd --permanent --service=isotovideo --add-port=$((i * 10 + 20003))/tcp ; done
firewall-cmd --permanent --zone=public --add-service=isotovideo
firewall-cmd --reload
# Test looks good!

1. Set up OpenVSwitch with MM networking:

systemctl enable --now openvswitch os-autoinst-openvswitch
# Copy openqa-prepare-mm-setup from ow19
./openqa-prepare-mm-setup
# Note: On ow20 it's eth1, not eth0
cat > /etc/firewalld/zones/trusted.xml <<EOF
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <interface name="br1"/>
  <interface name="eth1"/>
  <masquerade/>
</zone>
EOF
firewall-cmd --permanent --zone=public --remove-interface=eth1
firewall-cmd --reload
reboot

1. Wait until it's rebooted and log in again. The logs show that firewalld broke, because of an already fixed bug, so run zypper dup
2. Clone a MM test (firewalld-policy-firewall): openqa-clone-job --within-instance https://openqa.opensuse.org --max-depth 0 --skip-chained-deps 2928052 _GROUP=38 WORKER_CLASS=openqaworker20
3. Notice that it failed because ifcfg-eth1 still had ZONE=public.

nano /etc/sysconfig/network/ifcfg-eth1 /etc/firewalld/zones/public.xml /etc/firewalld/zones/trusted.xml
rcfirewalld restart

1. Noticed that the wicked script is invalid. Fix openqa-prepare-mm-setup (remove newline before #!/bin/sh, $bridge -> \$bridge) and run it again.
2. Rebooted and restarted failed test -> green

Setting due date based on mean cycle time of SUSE QE Tools

ow20 is enabled for production now, with tap. So far I didn't observe any issues.

I noticed that systemctl enable --now openqa-reload-worker-auto-restart@{1..20}.path was missing, did that as well.

at least I found "auto-update" missing from systemd timers. We should closely check if there is anything else missing compared to other o3 workers

favogt wrote:

okurz wrote:

at least I found "auto-update" missing from systemd timers. We should closely check if there is anything else missing compared to other o3 workers

I installed that - did I forget to enable it?

I have not found the package "openQA-auto-update" installed. I installed and enabled it with:

ssh root@openqaworker20 "zypper -n in openQA-auto-update && systemctl enable --now openqa-auto-update.timer"

Rest looks fine. Thanks for the good setup!