QA » openQA Project » openQA Tests

https://progress.opensuse.org/issues/70957 tracker issue in jump space

http://download.opensuse.org/repositories/openSUSE:/Maintenance:/Test:/Jump_15.2/ is the merged testrepo

Hi @msmeissn,

Right now there are these runs :

• https://openqa.opensuse.org/group_overview/75
• https://openqa.opensuse.org/group_overview/76

So what more is it expected from the openqa tests in relation to these repos? Is it sufficient for the system to be up-to-date, by using the aforementioned repositories and then do the testing?
thanks!

These are the groups for product development before release.

What we also need is the openqa setup for the maintenance phase.

e.g. for Leap 15.2: https://openqa.opensuse.org/group_overview/69 and https://openqa.opensuse.org/group_overview/70

Thank you @msmeissn for the answers!

After some research though I did today, an implementation of the openQA bot is also required which is handled by @osukup (who has time-off this week). Consequently, this ticket can be handled on the beginning of next week.

@tjyrinki_suse , it is an urgent+sprint item, but I have to postpone it for next week, will it be ok? :)

Sure ok @punkioudi, it's urgent but we need Ondrej indeed.

So what I get from asking around:

• Iso has to be synced (https://github.com/os-autoinst/openqa-trigger-from-obs/pull/89), but rather we need the fixed asset (that would be living in /var/lib/openqa/share/factory/hdd/fixed
• Medium type setup: (flavor=DVD-Incidents) in o3 for jump (Leap 15.2 mediums can be used as a template)
• It's not clear to me, But likely there's a go: Jump has to be added here: https://github.com/openSUSE/openSUSE-release-tools/pull/2383

See linked tickets for future reference

I've pinged osukup about this.

for schedule -> https://github.com/openSUSE/openSUSE-release-tools/tree/master/data

it is called leap now. jump was only for 15.2.1, but 15.3 will be called leap, also in the directory paths.

• How to test this?
• Who consumes/what the json files modified in the PR
• How do we check that the results are correct? (i.E that maintenance updates will be picked up)
• What does the syncing of the jobgroups between Maintenance and OBS once the leap product is released

someone needs to setup 15.3 job groups in openqa.opensuse.org. (antonios?)

There will be code changes needed in opensuse-release-tools and os-autoinst-distri-opensuse, as we now have 2 different looking incidents.

what is not handled yet, are incidents like this:

openSUSE:Maintenance:15953 hello

As we now have 2 sources for updates, openSUSE:Leap:15.3:Update AND openSUSE:Backports:SLE-15-SP3:Update to test for Leap 15.3.

openSUSE:Maintenance:15681 is the other possible case, with rpmlint-mini

BOTH cases must be handled.

The code changes in the repos have been done, and the mediums and JobGrous have been set. Temporarily the beta iso is being used.

I expected that they woulkd have been triggered. I'll investigate why they haven't.

The runs are now posted for "regular" updates but the bot continues to ignore the backports. I just asked the UM if it's possible to send a dummy update to the sle repo so that can be tested.

openSUSE:Maintenance:15953 - recommended (low): hello
locked: 4
release: 879946, review, 39 days ago

maintenance_release: openSUSE:Maintenance:15953/hello.openSUSE_Backports_SLE-15-SP3_Update -> openSUSE:Backports:SLE-15-SP3:Update/hello.15953
maintenance_release: openSUSE:Maintenance:15953/patchinfo -> openSUSE:Backports:SLE-15-SP3:Update/patchinfo.15953

It was missing a qam-openqa review, I added one just now.

The configuration for the packages specific to Leap 15.3 has been done and has been going on for some time. The problem is doing the rest of them. Complicating that, is, that to our knowledge is qe-core, the only documentation we have is the code, which means we just can't copy paste the config like we did before.

okurz wrote:

It is expected that QE-Core takes care of that. In best case this can be fixed with just configuration changes for the openSUSE Maintenance openQA bot, in worst case this might need more changes https://github.com/openSUSE/openSUSE-release-tools/blob/master/openqa-maintenance.py . msmeissn has reminded of the release date of Leap 15.3 in just a couple of days until this issue has been fixed

Ok.

1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. @hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

tjyrinki_suse wrote:

1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. @hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

The main and only reason to use openqa.suse.de is because the unreleased SLE maintenance updates are not available on openqa.opensuse.org. This should not be the default for new tests that run on Leap as Leap is still a community project which should be developed and tested in the open where possible.

how about starting with a single scenario as proof of concept? Especially as there would be no help from any openSUSE community to review and as people are already suffering from the review workload the additional tests should be really limited to only show that the SLE update side of things is not completely broken while keeping the main test load still on o3, right?

Today there was a meeting with Marcus Meissner, Antonios, Santi, Anna and myself with this topics:

• Anton has finished the set-up of the channels for 15.3, all good
• For 15.4 Lubos will create a progress ticket at Beta1 to inform QE that openQa needs to be set-up for SP4
• The review of the Leap maintenance job groups is still under discussion, Santi will follow-up with an email, the idea is that the QE Core team will take over the review -> more info will come
• Anna & QE Core team will maintain the tests for Leap and increase the test coverage step by step

waiting for apappas and osukup regarding scheduling tests within OSD. Would it better if we can not wait for 1:1 talks but better coordinate on team level. Maybe I can offer some help?

There was no update and we are in the dark here. We should have apappas provide the actual requirements.

Thanks for the update, Oli. But could you or somebody else please specify the "help" that Anton is waiting for? Is it the tools team, is it Ondrey Sukup or who else? On Wednesday in the call I had the impression that we are waiting for Anton. He has said that he can not solve the problem so we are going in cycles here.

I am waiting for Ondrej or people that have knowledge of the openqa maintenance bot to help me edit the ΜR(1) so that it can post the jobs.

1. https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/503

I commented in the MR: https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/503/diffs#note_335709

apappas wrote:

I am waiting for Ondrej or people that have knowledge of the openqa maintenance bot to help me edit the ΜR(1) so that it can post the jobs.

1. https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/503

Ondrej can you help to move this forward?

We discussed this in the weekly SUSE QE Sync 2021-08-04. Responsibility for qa-maintenance/openQABot itself is for sure within the scope of SUSE QE Tools. But enabling "openSUSE repo tests as part of SLE updates" is certainly a new domain and not something we have experience with. Just don't expect too much :)

I am not aware of the current limits of the bot, but I want to make it clear that what is needed is, to take incidents from the SUSE:Updates:openSUSE-SLE:15.3 internal build service project and post the corresponding jobs for an already set up OSD JobGroup. AFAICT this is already what it does for the SLE projects.

apappas wrote:

I am not aware of the current limits of the bot, but I want to make it clear that what is needed is, to take incidents from the SUSE:Updates:openSUSE-SLE:15.3 internal build service project and post the corresponding jobs for an already set up OSD JobGroup. AFAICT this is already what it does for the SLE projects.

OK, any example of incident with populated openSUSE-SLE:15.3 ?

• we now doing switch to shedule based on qem-dashboard with different format of config files .. can be found in qa-maintenance/metadata/bot-ng

SUSE:Maintenance:20230:245273
e.g. is a dracut update with openSUSE-SLE-15.3 in it.

https://smelt.suse.de/overview/ -> Testing -> search for openSUSE in the products.

msmeissn wrote:

SUSE:Maintenance:20230:245273
e.g. is a dracut update with openSUSE-SLE-15.3 in it.

https://smelt.suse.de/overview/ -> Testing -> search for openSUSE in the products.

thx

openSUSE-SLE has a different repo structure in IBS so it needs add few workaround into bot:(

https://gitlab.suse.de/osukup/bot-ng/-/commit/5e5c9ea1c025fc22fc2d12e349359541da0d01c2 and https://gitlab.suse.de/qa-maintenance/metadata/-/blob/master/bot-ng/leap153.yml done ..

jobs are scheduled in osd, but there is a missing Leap 15.3 ISO and the second question is if Leap needles are deployed for OSD jobs

Discussed in weekly QE sync 2021-09-01: This ticket was mentioned again by multiple stakeholders and should be worked on as soon as apappas is available again. @apappas if you need support from members of SUSE QE Tools please reach out to us. So far I see the prerequisites in place. Of course test scheduling details need to be ruled out.

Waiting for Anton to come back next monday

@apappas seems like you enabled broken Leap tests in production job groups so we have scenarios that "never worked" affecting releases of updates. Please do not do something like this. For example there is https://openqa.suse.de/tests/7142084#next_previous

Can you revert this ASAP and/or move tests to the "development" job group to first stabilize before moving into the production job groups?

EDIT: Wait, they are in the development job group like https://openqa.suse.de/tests/overview?distri=opensuse&version=15.3&build=%3A21237%3Asamba&groupid=393 . I think I accused you wrongly, sorry.

apappas wrote:

IMHO the group is setup, the scenarios need some love by the various teams (needles etc.) and the schedule needs defining but I think that's a topic for another ticket. I am asking for feedback to resolve this ticket and make new ones.

We discussed this in weekly QE sync 2021-09-22 and clarified: @apappas yes, please create new tickets (first!) assigned to the according squads as further subtasks of #92563 and then (afterwards!) resolve this ticket.

Email mentioned at https://progress.opensuse.org/issues/72202#note-59 has been finally sent: https://mailman.suse.de/mailman/private/qa-team/2021-September/005466.html TL;DR: go here and check: https://confluence.suse.com/display/qasle/Tests+Maintained+by+QE+Core#TestsMaintainedbyQECore-openqa.opensuse.org

I wonder if it's desired to have also kernel testing for Leap 15.3. Just out of curiosity, because kernel QE team is that busy, that even Tumbleweed is not reviewed and tagged regularly (I try to watch it, but SLE and other projects are the priority).

pvorel wrote:

I wonder if it's desired to have also kernel testing for Leap 15.3. Just out of curiosity, because kernel QE team is that busy, that even Tumbleweed is not reviewed and tagged regularly (I try to watch it, but SLE and other projects are the priority).

I consider the risk of breaking Leap with SLE kernel updates low considering that with Leap 15.3 after the "Jump" project the very same SLE binaries are used in Leap. Implicitly of course the kernel is tested against Leap with the existing Leap (for SLE maintenance updates) tests. As you said you are too busy so better focus on Tumbleweed tests with the more recent kernel versions before considering extending Leap kernel specific tests.

the kernel opensuse leap 15.3 uses is tested by the SLE kernel testing... so I think it does not need it seperately for openSUSE at this time.

Is the status of this ticket current? any updates? can we resolve the ticket?

So the opensuse.openqa.org side this was done and is running in production.

For the IBS side we talked about it, but without a link to the openqa group where it happens I do not consider it done.

I looked into "Core Maintenance" runs, it seems not to be in.