QA » openQA Project » openQA Tests

but at least https://openqa.opensuse.org/tests/1464522#step/await_install/69 seemed to have progressed. Do you have an idea why this worked then?

okurz wrote:

but at least https://openqa.opensuse.org/tests/1464522#step/await_install/69 seemed to have progressed. Do you have an idea why this worked then?

You have already fixed the S390_NETWORK_PARAMS where the four machines where trying to use the same IP address. That may be enough. I will check.

In https://openqa.opensuse.org/tests/1465662#step/bootloader_s390/49 I could see the error when trying to access openSUSE-Tumbleweed-oss-s390x-Snapshot20201106/suse.ins . The repo folder does not exist (anymore?) on o3. Instead there is openSUSE-Tumbleweed-oss-s390x-Snapshot20201108/ by now. Seems like the quota is exceeded. I crosschecked the source and debuginfo folders but they are not taking up too much:

okurz@ariel:~> du -sh /var/lib/openqa/share/factory/repo/openSUSE-Tumbleweed-oss-s390x-Snapshot20201108*
55G /var/lib/openqa/share/factory/repo/openSUSE-Tumbleweed-oss-s390x-Snapshot20201108
116K    /var/lib/openqa/share/factory/repo/openSUSE-Tumbleweed-oss-s390x-Snapshot20201108-debuginfo
36M /var/lib/openqa/share/factory/repo/openSUSE-Tumbleweed-oss-s390x-Snapshot20201108-source

The current assets size limit for the job group "openSUSE Tumbleweed s390x" is 80GB meaning that only a single snapshot can be kept. As we have an okaish storage situation on o3 right now I increased the size limit now to 240GB so that 4 snapshots could be kept.

another job https://openqa.opensuse.org/tests/1465663#step/bootloader_s390/29 looks like it is trying to reach an ftp folder that has its name truncated. Be aware about the line length limit in xedit, see https://openqa.opensuse.org/tests/1465663#step/bootloader_s390/28 where the first entry line is wrapped. Likely these characters are lost. I suggest you try to replace the IPv4 address of 192.168.112.100 with a hostname for the openQA webUI host.

Yes, the problem still happens: https://openqa.opensuse.org/tests/1465663#step/bootloader_s390/30
And now I can reproduce it with all z/VM guests.

The funny thing is that it was working with linux146: https://openqa.opensuse.org/tests/1464522#step/bootloader_s390/28
I thought on the cropping but suspected something related to the guest profile or Xedit profile. I am going to apply the same profiles we use in the z/VM guests for SLE.

I have already set the guest profile and Xedit profile to behave better on linux144, but performing a manual installation it also cannot reach the repository, not even http://download.opensuse.org/tumbleweed/repo/oss/

I have also tried with Layer2=0, because I had this problem with one machine once, but same error.

I am thinking that the Channels are wrong (ReadChannel=0.0.0800 WriteChannel=0.0.0801 DataChannel=0.0.0802), which also happened to me once. I need to ask Ihno.

Also, the z/VM guest can resolve openqa to 1620:113:80c0:8080:10:160:0:207, but it is not able to ping an IPv6 address DTCPIN0029E SendTo(): EDC8118I Network is unreachable. (errno2=0x000005DF)

Can it be, that it is a VLAN issue? The last successful step is the VLAN configuration...
https://openqa.opensuse.org/tests/1465663#step/bootloader_s390/30

AdaLovelace wrote:

Can it be, that it is a VLAN issue? The last successful step is the VLAN configuration...
https://openqa.opensuse.org/tests/1465663#step/bootloader_s390/30

Yes, it could be.
I am asking Yast developers about debugging linuxrc to know what is happening in the SUT.

To strike one item from the list, the four z/VM guests have the expected profile exec a and profile xedit a by openQA.
I keep a copy of the templates in:

• https://github.com/binary-sequence/linux-on-the-ibm-mainframe/blob/development/zVM/profile.xedit.rexx
• https://github.com/binary-sequence/linux-on-the-ibm-mainframe/blob/development/zVM/profile.exec.rexx

While trying to setup a machine to also be able to gather the linuxrc logs I found some errors that explains the network problem.
It was always failing in the same spot, but for different reasons.
One problem was that I didn't know the role that Hostname plays for openQA. Before there was a machine, now there are more and each one need a proper setting, that matches /etc/hosts from ariel. (that okurz added)
Then, I had to also add those to /etc/hosts of the worker (rebel).
Then, the failing job was still using the IP from the old broken z/VM guest. okurz fixed the settings in openQA, but I made the mistake of cloning the job without overriding the new IP for the new 4 z/VM guests.
And finally the problem of a very long URL that overflows the max line length in the parmfile. Each line of the parmfile is a virtual punched card, physicaly max 80, but 73-80 was used as comments and therefore ignore by the reader. I don't know why it was working before. I assume there is a setting I don't know that can regulate this max line length and in those 4 machines is 72.
Verification run using a symbolic link in ariel to make the directory repo shorter: https://openqa.opensuse.org/tests/1466408#step/bootloader_s390/34

So, I think it is time now to implement my idea of info files. Other option would be to tell rsync.pl to create symbolic links with shorter names for s390 repositories. But, anyway info file requires less typing from openQA, therefore less typing issues on bootloader.

IPv6 sounds like a good idea. Can we provide an IPv6 address to z/VM guests with according configuration? Feel free to (carefully) apply changes to o3 or rebel where needed to make sure IPv6 resolution works over these steps as well.

okurz wrote:

IPv6 sounds like a good idea. Can we provide an IPv6 address to z/VM guests with according configuration? Feel free to (carefully) apply changes to o3 or rebel where needed to make sure IPv6 resolution works over these steps as well.

AFAIK there is not involvment in rebel or the Linux SUT, they can both resolve IPv6, the problem are the z/VM guests. And that setup has to be done by Ihno I think.

Ihno has clarified me some things:

you can have lines longer than 80 character in a parmfile.
When the parmfile is handed over to the kernel all lines
are just concatenated.
When a line has 80 characters and in the next line
characters are in the first column, and both lines are
concatenated you have a longer line.

At no time (even the editing in z/vm) there should be
things added to the end of the line.

The parmfile is not allowed to have more than 8 lines.
Everything else will get lost/truncated.
I found a file on linux144 (linux144.parmfile) where
every parameter was in one line. This will not work.

= parmfile with lines longer than 80 characters =
So, that resolves the question of why it was working before even if the parameter was longer than 80 lines.
The question is why openQA is not able anymore to properly split that line now?

= parmfile line 8 line limit =
good to know! I was frustrated because I wasn't able to have the manual installation running while openQA could.

The info file option doesn't work if the info file is provided from the worker because the URL is very long and the URL shortener used is only reachable inside suse.de: https://openqa.opensuse.org/tests/1467731#step/bootloader_s390/26

I also tried using v.gd, but it fails: https://openqa.opensuse.org/tests/1467718#step/bootloader_s390/6

I also tried the concatenation of the parmfile that Ihno told me.

The limit is 72 characters.
The character 73 is the first one in the month 11, and it gets lost: https://openqa.opensuse.org/tests/1467778#step/bootloader_s390/28
If I use 72 characters the parmfile looks ok: https://openqa.opensuse.org/tests/1467779#step/bootloader_s390/28
But, linuxrc still gets the URL cropped: https://openqa.opensuse.org/tests/1467779#step/bootloader_s390/29
I should ask Yast team tomorrow.

The two options left are IPv6 in the z/VM guests (Ihno wanted to look into it) and short symlink by rsync.pl.

I didn't have time today to ask Ihno about his progress with IPv6 on the z/VM guest nor the Yast team for the unexpected behavior of parmfile->linuxrc

I am looking https://github.com/os-autoinst/openqa-trigger-from-obs to see if I can teach it to create short symlinks to the s390x repos.

After talking to https://github.com/andrii-suse, he suggested as simple solution to do the following change: https://github.com/os-autoinst/openqa-trigger-from-obs/pull/107

PR merged and deployed, next Snapshot will use shorter name.

I fear you should clone it to openSUSE to be public :(

okurz wrote:

I fear you should clone it to openSUSE to be public :(

https://bugzilla.opensuse.org/show_bug.cgi?id=1178790 :)