QA » openQA Project » openQA Infrastructure

I'm bumping prio since the machine is completely offline right now and not just outdated. I made this clear in the title also. And Blocked with a due date on Thursday on the off chance that we forget about it.

cdywan wrote:

Seems like there's several redundant disk errors on the machine that look like this and it's currently stuck in grub:

BTRFS error (device) md0p1): Remounting read0write after error is not allowed

@osukup reached out to @mrueckert to get access to the console

Moving due date as per conversation in chat since we're waiting on other people and it's not considered super urgent.

@nicksinger the infra ticket was resolved on 2021-09-07, so did you check if you do have IPMI access or something?

unfortunately I didn't receive notifications. The ticket was closed with "please open a jira SD ticket". Done so now: https://sd.suse.com/servicedesk/customer/portal/1/SD-60360 (can anybody see this besides me?)

Nothing else happend.

Thanks! I can see it now. Bumping the due date to Wednesday for now.

I could actually access the machine now over IPMI (see SD ticket). I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/358

The SD ticket is still open and I asked for a proper DNS entry for IPMI. However, with the current state it should be possible to proceed hence "unblocking".

we could assign a *.qa.suse.de domain if nothing happens in the ticket.

To be checked on-site at next possibility

@nicksinger as discussed in daily as you could not find anyone in office to collaborate on please create EngInfra ticket stating what goal we want to reach, e.g. check with IPMI command that we can read what's going on in the serial terminal, and suggest to change UEFI settings or something "on-site"

setting much further due-date as we rely a lot on individuals present on-site in nbg office and imagetester turned out to be not that critical right now, especially as we found already workarounds how to run openQA workers in containers (for s390x but we could apply the same elsewhere when we need to). Please still act urgently on raising the request with EngInfra, then we wait

Alright, after talking to @mkittler today after the javaws stuff I had another idea how to access the iKVM of that machine. I went to the web interface of the BMC at http://10.160.65.195 and clicked on the "Remote Console Preview" image there. This offers you to download a launch.jnlp file. Executing it on the console unfortunately fails with:

nsinger@workstation ~/Downloads » LANG=C javaws launch.jnlp
selected jre: /etc/java-config-2/current-system-vm/jre/
Warning!, Fall back in resolve_jar to hardcoded paths:
no
selected jre: /etc/java-config-2/current-system-vm/jre/
Warning!, Fall back in resolve_jar to hardcoded paths:
no
You are trying to get resource https://10.160.65.195:443/iKVM.jar but it is not in cache and could not be downloaded. Attempting to continue, but you may expect failure
You are trying to get resource https://10.160.65.195:443/liblinux_x86_64.jar but it is not in cache and could not be downloaded. Attempting to continue, but you may expect failure
JAR https://10.160.65.195:443/iKVM.jar not found. Continuing.
JAR https://10.160.65.195:443/liblinux_x86_64.jar not found. Continuing.
JAR https://10.160.65.195:443/iKVM.jar not found. Continuing.
JAR https://10.160.65.195:443/liblinux_x86_64.jar not found. Continuing.
netx: Initialization Error: Could not initialize application. (Fatal: Initialization Error: Unknown Main-Class. Could not determine the main class for this application.)
net.sourceforge.jnlp.LaunchException: Fatal: Initialization Error: Could not initialize application. The application has not been initialized, for more information execute javaws from the command line.
    at net.sourceforge.jnlp.Launcher.createApplication(Launcher.java:822)
    at net.sourceforge.jnlp.Launcher.launchApplication(Launcher.java:531)
    at net.sourceforge.jnlp.Launcher$TgThread.run(Launcher.java:945)
Caused by: net.sourceforge.jnlp.LaunchException: Fatal: Initialization Error: Unknown Main-Class. Could not determine the main class for this application.
    at net.sourceforge.jnlp.runtime.JNLPClassLoader.initializeResources(JNLPClassLoader.java:774)
    at net.sourceforge.jnlp.runtime.JNLPClassLoader.<init>(JNLPClassLoader.java:338)
    at net.sourceforge.jnlp.runtime.JNLPClassLoader.createInstance(JNLPClassLoader.java:421)
    at net.sourceforge.jnlp.runtime.JNLPClassLoader.getInstance(JNLPClassLoader.java:495)
    at net.sourceforge.jnlp.runtime.JNLPClassLoader.getInstance(JNLPClassLoader.java:468)
    at net.sourceforge.jnlp.Launcher.createApplication(Launcher.java:814)
    ... 2 more

Following the leads in the error message I watched at an strace of the same "Remote console" but from openqaworker8. I grabbed its liblinux_x86_64.jar and iKVM.jar in the hopes that it might somehow work. It didn't. But the local webserver showed me what files imagetesters BMC actually requested:

2021-10-05 13:04:33.991 [INFO ] [::ffff:127.0.0.1]:35580 - HEAD /liblinux_x86_64__V1.0.3.jar.pack.gz (local: ./liblinux_x86_64__V1.0.3.jar.pack.gz)
2021-10-05 13:04:33.991 [INFO ] [::ffff:127.0.0.1]:35582 - HEAD /iKVM__V1.69.13.0x0.jar.pack.gz (local: ./iKVM__V1.69.13.0x0.jar.pack.gz)

With this information I was finally able to request the original files of from imagetester and put them into a temporary directory:

curl -k https://10.160.65.195:443/liblinux_x86_64__V1.0.3.jar.pack.gz > /tmp/ikvm/liblinux_x86_64__V1.0.3.jar.pack.gz
curl -k https://10.160.65.195:443/iKVM__V1.69.13.0x0.jar.pack.gz > /tmp/ikvm/iKVM__V1.69.13.0x0.jar.pack.gz

I then started a webserver serving these files and modified the launch.jnlps first line from

<jnlp spec="1.0+" codebase="https://10.160.65.195:443/">

to:

<jnlp spec="1.0+" codebase="http://127.0.0.1:8888/">

Now I can start this modified jnlp-file with javaws launch.jnlp and it opens up a java application where I'm able to see the graphical output of imagetester. I can now start the investigation and we're not blocked by infra anymore.

I booted a systemrescuecd and scrubbed the btrfs filesystem on the disk: no issues reported. Smart values look fine but error reporting doesn't seem to be supported so I couldn't check if there was any error reported in the past.
I then chrooted into the system (following https://wiki.gentoo.org/wiki/Chroot/en#Configuration):

chroot /mnt/mychroot
mount -a
transactional-update shell
zypper ref
zypper up

but it failed with:

Download (curl) error for 'http://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.2/x86_64/os-autoinst-4.6.1633339717.7d37d2ac-lp152.875.1.x86_64.rpm':
Error code: Curl error 60
Error message: SSL certificate problem: self signed certificate in certificate chain

(all previous 227 packages where fine and no such error came up). I then tried to reboot the machine into the live-system again and it came back up. Not sure if the scrub fixed it or if transactional-upgrade did some magic to make it work again. On the live system I gave it another try with transactional-update up which successfully updated all installed packages. I then did another reboot and the machine came up successfully without any failing services in systemd. transactional-update.timer is also running too so lets see if the machine now works again.

Did a transactional-update dup reboot, updates where successful and a reboot got scheduled:

imagetester:~ # rebootmgrctl is-active
RebootMgr is active
imagetester:~ # rebootmgrctl get-strategy
Reboot strategy: best-effort
imagetester:~ # rebootmgrctl get-window
Maintenance window is set to *-*-* 03:30:00, lasting 01h30m.
imagetester:~ # rebootmgrctl status
Status: Reboot requested, waiting for maintenance window

I canceled that reboot and triggered an immediate one: rebootmgrctl reboot now. After a few seconds the machine was back up and running.