QA » openQA Project » openQA Tests

https://progress.opensuse.org/issues/70957 tracker issue in jump space

http://download.opensuse.org/repositories/openSUSE:/Maintenance:/Test:/Jump_15.2/ is the merged testrepo

Hi msmeissn,

Right now there are these runs :

• https://openqa.opensuse.org/group_overview/75
• https://openqa.opensuse.org/group_overview/76

So what more is it expected from the openqa tests in relation to these repos? Is it sufficient for the system to be up-to-date, by using the aforementioned repositories and then do the testing?
thanks!

These are the groups for product development before release.

What we also need is the openqa setup for the maintenance phase.

e.g. for Leap 15.2: https://openqa.opensuse.org/group_overview/69 and https://openqa.opensuse.org/group_overview/70

Thank you msmeissn for the answers!

After some research though I did today, an implementation of the openQA bot is also required which is handled by osukup (who has time-off this week). Consequently, this ticket can be handled on the beginning of next week.

tjyrinki_suse , it is an urgent+sprint item, but I have to postpone it for next week, will it be ok? :)

Sure ok punkioudi, it's urgent but we need Ondrej indeed.

So what I get from asking around:

• Iso has to be synced (https://github.com/os-autoinst/openqa-trigger-from-obs/pull/89), but rather we need the fixed asset (that would be living in /var/lib/openqa/share/factory/hdd/fixed
• Medium type setup: (flavor=DVD-Incidents) in o3 for jump (Leap 15.2 mediums can be used as a template)
• It's not clear to me, But likely there's a go: Jump has to be added here: https://github.com/openSUSE/openSUSE-release-tools/pull/2383

See linked tickets for future reference

I've pinged osukup about this.

for schedule -> https://github.com/openSUSE/openSUSE-release-tools/tree/master/data

it is called leap now. jump was only for 15.2.1, but 15.3 will be called leap, also in the directory paths.

• How to test this?
• Who consumes/what the json files modified in the PR
• How do we check that the results are correct? (i.E that maintenance updates will be picked up)
• What does the syncing of the jobgroups between Maintenance and OBS once the leap product is released

someone needs to setup 15.3 job groups in openqa.opensuse.org. (antonios?)

There will be code changes needed in opensuse-release-tools and os-autoinst-distri-opensuse, as we now have 2 different looking incidents.

what is not handled yet, are incidents like this:

openSUSE:Maintenance:15953 hello

As we now have 2 sources for updates, openSUSE:Leap:15.3:Update AND openSUSE:Backports:SLE-15-SP3:Update to test for Leap 15.3.

openSUSE:Maintenance:15681 is the other possible case, with rpmlint-mini

BOTH cases must be handled.

The code changes in the repos have been done, and the mediums and JobGrous have been set. Temporarily the beta iso is being used.

I expected that they woulkd have been triggered. I'll investigate why they haven't.

The runs are now posted for "regular" updates but the bot continues to ignore the backports. I just asked the UM if it's possible to send a dummy update to the sle repo so that can be tested.

openSUSE:Maintenance:15953 - recommended (low): hello
locked: 4
release: 879946, review, 39 days ago

maintenance_release: openSUSE:Maintenance:15953/hello.openSUSE_Backports_SLE-15-SP3_Update -> openSUSE:Backports:SLE-15-SP3:Update/hello.15953
maintenance_release: openSUSE:Maintenance:15953/patchinfo -> openSUSE:Backports:SLE-15-SP3:Update/patchinfo.15953

It was missing a qam-openqa review, I added one just now.

The configuration for the packages specific to Leap 15.3 has been done and has been going on for some time. The problem is doing the rest of them. Complicating that, is, that to our knowledge is qe-core, the only documentation we have is the code, which means we just can't copy paste the config like we did before.

okurz wrote:

It is expected that QE-Core takes care of that. In best case this can be fixed with just configuration changes for the openSUSE Maintenance openQA bot, in worst case this might need more changes https://github.com/openSUSE/openSUSE-release-tools/blob/master/openqa-maintenance.py . msmeissn has reminded of the release date of Leap 15.3 in just a couple of days until this issue has been fixed

Ok.

1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

tjyrinki_suse wrote:

1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

The main and only reason to use openqa.suse.de is because the unreleased SLE maintenance updates are not available on openqa.opensuse.org. This should not be the default for new tests that run on Leap as Leap is still a community project which should be developed and tested in the open where possible.

how about starting with a single scenario as proof of concept? Especially as there would be no help from any openSUSE community to review and as people are already suffering from the review workload the additional tests should be really limited to only show that the SLE update side of things is not completely broken while keeping the main test load still on o3, right?

Today there was a meeting with Marcus Meissner, Antonios, Santi, Anna and myself with this topics:

• Anton has finished the set-up of the channels for 15.3, all good
• For 15.4 Lubos will create a progress ticket at Beta1 to inform QE that openQa needs to be set-up for SP4
• The review of the Leap maintenance job groups is still under discussion, Santi will follow-up with an email, the idea is that the QE Core team will take over the review -> more info will come
• Anna & QE Core team will maintain the tests for Leap and increase the test coverage step by step

waiting for apappas and osukup regarding scheduling tests within OSD. Would it better if we can not wait for 1:1 talks but better coordinate on team level. Maybe I can offer some help?