QA » openQA Project » openQA Tests

[26/02/2019 15:45:16] <SergioAtSUSE> okurz, then here. There is already a built Tumbleweed IDO for s390x than can be installed on a nested ZVM/KVM. We want to start performing tests on O3. We need to have the assets synced there. Do you have access to the script that syncs Tumbleweed?
[26/02/2019 15:45:22] <SergioAtSUSE> ISO*
[26/02/2019 15:46:18] <okurz> SergioAtSUSE: every SUSE employee has access. And there are already tests enabled for s390x. You don't need to "start" anything but just make sure the media are released to the according :ToTest repo
[26/02/2019 15:47:17] <okurz> SergioAtSUSE: https://gitlab.suse.de/openqa/scripts/blob/master/rsync_opensuse.pm#L72
[26/02/2019 15:49:29] <okurz> SergioAtSUSE: you can compare https://build.opensuse.org/package/show/openSUSE:Factory/000product with https://build.opensuse.org/package/show/openSUSE:Factory:zSystems:ToTest/000product
[26/02/2019 15:50:52] <okurz> SergioAtSUSE: let me check the current sync status on o3
[26/02/2019 15:56:01] <okurz> SergioAtSUSE: hm, actually seems like the assets are more or less fine but "ERROR: destination must be a directory when copying more than 1 file"
[26/02/2019 15:56:49] <SergioAtSUSE> okurz, where have you got that error?
[26/02/2019 15:57:03] <okurz> SergioAtSUSE: from o3
[26/02/2019 16:01:27] <SergioAtSUSE> okurz, could you create the missing directory on O3?
[26/02/2019 16:52:25] <okurz> DimStar: maybe you can crosscheck nevertheless. rsync://openqa@obs-backend.publish.opensuse.org/opensuse-internal/build//openSUSE:Factory:zSystems:ToTest/images//local/*product:openSUSE-ftp-ftp-s390x/openSUSE-*-s390x-Media1/media.1/media should be right to read out build information, right?
[26/02/2019 16:54:51] <okurz> yes, that could help
[26/02/2019 16:57:08] <DimStar> okurz: cleaned; let's give OBS a couple minutes then retry.. maybe that's already all we need
[26/02/2019 17:02:56] <okurz> DimStar: seems like that helped already, sync started

currently syncing on o3

coolo is ok to deactivate apparmor if there is no other service running on that machine (IT policy for company public services).

• imagetester is a worker for x86_64 and it is not possible to disable apparmor.
• remote-backend workers ssh into the wild, which cannot be properly confined in apparmor

Proposed solution, we need to find a dedicated machine for remote-backend workers for openqa.opensuse.org.

SLindoMansilla wrote:

• imagetester is a worker for x86_64 and it is not possible to disable apparmor.

why should it be not possible?

Trying to setup a Raspberry Pi as worker for svirt. (with the intention to put it on the O3 network if the experiment works)

I was able to set up a Raspberry Pi 3 B+ as an openQA-worker for remote z/VM SUT's host.

Internal links:

• Successful SLE15-SP1 installation test: http://openqa.slindomansilla-vm.qa.suse.de/tests/1240
• Request for adding the Raspberry Pi inside the O3 workers network is issued: https://infra.nue.suse.com/SelfService/Display.html?id=132506
• In the meantime, trying to get the openSUSE Tumbleweed installation working: http://openqa.slindomansilla-vm.qa.suse.de/tests/1241

After analyzing the situation with R&D admins, the server room that is on the backend for openqa.opensuse.org has a strict policy about only allowing certified machines.
Raspberry Pi is not a valid machine.

About getting a certified machine for that server room:
After talking to the openSUSE chairman, the existing machines are donated to openSUSE by SUSE, so, I will ask my department if we have the budget for that.

Machine requirement from R&D admins for server room 1: "serverlike shape, so 19" size 1 or 2 HE 1-2 powerplugs and normally a BMC port"

azouhr suggested this machine: https://www.ebay.de/itm/Supermicro-SuperServer-502-200-X7SLA-H-Intel-Atom-330-1-60GHz-2GB-RAM-250G-HDD/223216622388?hash=item33f8bf5b34:g:MwMAAOSwa81aEtG9

R&D admins told me that such a machine would be valid.

Waiting for feedback from my department.

For several reasons, it is not allowed that the company buys hardware from ebay.

In this egg-chicken situation, that means that I need to adapt the backend to work with AppArmor enabled before enabling openSUSE tests on s390x.
For this, I need to be provided with a z/VM user that I can use for this purpose. Waiting for feedback from Z system admins.

@SLindoMansilla I think you found a different approach since your last ticket update, haven't you? We have "rebel" as an o3 worker. I am not sure how you want to control it. If you want I think we can include it and try to manage it just like the other o3 workers when you think this is the best approach. dmcvicar asked in https://chat.suse.de/channel/suse?msg=6H9kDMt4gGxc3tAYF about the status of Tumbleweed s390x and dimstar force-pushed a build of openSUSE:Factory:zSystems to o3. The jobs were scheduled but not triggered as there was no openQA worker instance running on rebel, the machine did not properly update since about 120 days. I fixed the update on rebel and rebooted into the new snapshot, the worker started correctly and started the job https://openqa.opensuse.org/tests/1054455 which failed in https://openqa.opensuse.org/tests/1054455#step/bootloader_s390/35 with an Input/output error on DASD device when trying to start installation. Maybe the DASD device previously assigned to the z/VM instance is now used elsewhere as well or needs to be re-initialized?

Nice, congratulations!
Can I add s390x LTP tests into kernel Tumbleweed tests https://openqa.opensuse.org/group_overview/32
I'd start with just 3 most important: install_ltp, ltp_syscalls and ltp_cve

pvorel wrote:

Nice, congratulations!
Can I add s390x LTP tests into kernel Tumbleweed tests https://openqa.opensuse.org/group_overview/32
I'd start with just 3 most important: install_ltp, ltp_syscalls and ltp_cve

Thanks!

More coverage is always better, but, could add them first to "Development Tumbleweed" https://openqa.opensuse.org/group_overview/38 ?
Once they are green or orange, no problem to add them to "PROD"

SLindoMansilla wrote:

More coverage is always better, but, could add them first to "Development Tumbleweed" https://openqa.opensuse.org/group_overview/38 ?
Once they are green or orange, no problem to add them to "PROD"

Sure, I'll add it first to development. Just a note, syscalls are always read, as that's a big collection of tests, very rarely they're all green even on SLES (very few tests fail, but of course still it's good to run them on both SLES and Tumbleweed).

pvorel wrote:

Just a note, syscalls are always read, as that's a big collection of tests, very rarely they're all green even on SLES (very few tests fail, but of course still it's good to run them on both SLES and Tumbleweed).

Then please don't add them. openQA should be used and is used by many as a validation platform, especially openqa.opensuse.org meaning that tests should only fail when they show new regressions that are also handled accordingly. We have limited ressources, especially on o3 and there are enough tests ignored and wasting ressources in "Development" already.

Oliver, kernel syscalls is the main validation for kernel, used on other the other archs. Failing tests are mostly on all archs. BTW we talk about 5 failing tests of 1304 fails (you know, kernel is never "fixed", there is always some error, thus stop testing because 0.5% of tests does not pass doesn't make sense :)).

you seem to have a different understanding of what a "test" is. IMHO, especially for openQA tests, every test result on top level is boolean, either it passed or not. If "not" then this means "block the release of the product that is tested" until resolved. Just keep in mind that openSUSE Tumbleweed and openSUSE Leap use a bot that releases when there are no not-ignored, failing test cases left, i.e. needs review of test failures and reliable results. Everything else has simply no place on o3 outside the "Development" job groups. And I hope to arrive at this approach for osd as well to get rid of the tedious very manual approach for test result reviewing.

Regards to failures: we're slowly working on fixing them (ongoing process), much bigger problem than few failures is #63373, as it breaks all kernel testing on intel. That's what really spoil the validation (and nobody cares).

These tests: we check the results (it's very useful to see bugs on Tumbleweed, which is close to mainline, but not the same). That's the only reason why I'm adding also s390x tests.

Added these 3 tests install_ltp, ltp_cve and ltp_syscalls into https://openqa.opensuse.org/group_overview/38. If they work ok, I'll move them to kernel (https://openqa.opensuse.org/group_overview/32) and add some more (just a few, as s390x is probably busy a lot).

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: textmode
https://openqa.opensuse.org/tests/1227549

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed