Project

General

Profile

action #72202

[qe-core] setup openSUSE Leap 15.3 openQA

Added by msmeissn about 1 year ago. Updated 21 days ago.

Status:
Resolved
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
2021-09-22
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

(this old ticket started as a ticket for Jump, later modified to be about Leap 15.3)

We need to setup openSUSE Leap 15.3 openqa.

It will have 3 update repos it consumes:

http://download.opensuse.org/update/leap/15.3/sle/
http://download.opensuse.org/repositories/openSUSE:/Backports:/SLE-15-SP3:/Update/standard/
http://download.opensuse.org/repositories/openSUSE:/Maintenance:/Test:/Leap_15.3/

the full test repo is at:

the incidents are below openSUSE Maintenance as usual, example incident openSUSE:Maintenance:14199

Current Build ISOs are below http://download.opensuse.org/distribution/leap/15.3/

[Acceptance Criteria]

  • Write also the documentation of the process preferably in confluence, see notes in comment #note-13

* Clarify why are we testing what on leap :)

This would need at least PR to https://github.com/openSUSE/openSUSE-release-tools/tree/master/data


Related issues

Related to openSUSE Leap 15.0 - action #25032: enable openQA maintenance testingResolved2018-04-012018-04-19

Related to openSUSE Leap 15.1 - action #39449: enable openQA maintenance testingResolved2019-03-112019-04-05

Related to openQA Tests - action #91827: [qe-core] Write tests and documentation for oqamaint botWorkable

Related to QA - action #96752: 'openSUSE-SLE' product schedule jobs to OSDResolved2021-08-11

Precedes openSUSE Leap 15.3 - action #76702: [qe-core][u] disable iso download in openQARejected2021-09-232021-09-23

History

#1 Updated by msmeissn about 1 year ago

https://progress.opensuse.org/issues/70957 tracker issue in jump space

#3 Updated by tjyrinki_suse about 1 year ago

  • Status changed from New to Workable
  • Priority changed from Normal to High
  • Start date deleted (2020-10-02)

#4 Updated by hrommel1 about 1 year ago

  • Due date set to 2020-10-30

Please evaluate until the deadline if there are any challenges with setting up.

Planned release is in November.

#5 Updated by geor about 1 year ago

  • Priority changed from High to Urgent

#6 Updated by geor about 1 year ago

  • Description updated (diff)

#7 Updated by geor about 1 year ago

  • Subject changed from setup openSUSE Jump 15.2 openqa to [sprint] setup openSUSE Jump 15.2 openqa

#8 Updated by punkioudi 12 months ago

  • Assignee set to punkioudi

#9 Updated by punkioudi 12 months ago

Hi msmeissn,

Right now there are these runs :

So what more is it expected from the openqa tests in relation to these repos? Is it sufficient for the system to be up-to-date, by using the aforementioned repositories and then do the testing?
thanks!

#10 Updated by msmeissn 12 months ago

These are the groups for product development before release.

What we also need is the openqa setup for the maintenance phase.

e.g. for Leap 15.2: https://openqa.opensuse.org/group_overview/69 and https://openqa.opensuse.org/group_overview/70

#11 Updated by punkioudi 12 months ago

Thank you msmeissn for the answers!

After some research though I did today, an implementation of the openQA bot is also required which is handled by osukup (who has time-off this week). Consequently, this ticket can be handled on the beginning of next week.

tjyrinki_suse , it is an urgent+sprint item, but I have to postpone it for next week, will it be ok? :)

#12 Updated by tjyrinki_suse 12 months ago

Sure ok punkioudi, it's urgent but we need Ondrej indeed.

#13 Updated by szarate 12 months ago

So what I get from asking around:

#14 Updated by szarate 12 months ago

  • Due date changed from 2020-10-30 to 2021-02-01
  • Priority changed from Urgent to Low

So after double checking with Marcus, this ticket is not urgent anymore as the Jump 15.2.1 received a no-go.

I'd like to announce that we have a final no-Go for Leap 15.2.1 after talking to stakeholders or engineers working on blocking issues.

https://lists.opensuse.org/opensuse-project/2020-11/msg00001.html

#15 Updated by msmeissn 12 months ago

  • Subject changed from [sprint] setup openSUSE Jump 15.2 openqa to [sprint] setup openSUSE Jump 15.3 openqa

change to be the 15.3 setup ticket.

#16 Updated by punkioudi 12 months ago

  • Subject changed from [sprint] setup openSUSE Jump 15.3 openqa to setup openSUSE Jump 15.3 openqa

I remove also the sprint tag for now. Thanks szarate and msmeissn for the update!

#17 Updated by punkioudi 12 months ago

  • Assignee deleted (punkioudi)

#18 Updated by szarate 12 months ago

  • Description updated (diff)

#19 Updated by szarate 12 months ago

  • Related to action #25032: enable openQA maintenance testing added

#20 Updated by szarate 12 months ago

  • Related to action #39449: enable openQA maintenance testing added

#21 Updated by szarate 12 months ago

See linked tickets for future reference

#22 Updated by tjyrinki_suse 12 months ago

  • Subject changed from setup openSUSE Jump 15.3 openqa to [qe-core] setup openSUSE Jump 15.3 openqa

#23 Updated by tjyrinki_suse 11 months ago

  • Project changed from QAM to openQA Tests
  • Priority changed from Low to Normal

#24 Updated by tjyrinki_suse 11 months ago

  • Category set to New test

#25 Updated by msmeissn 9 months ago

  • Subject changed from [qe-core] setup openSUSE Jump 15.3 openqa to [qe-core] setup Leap Jump 15.3 openqa

Name is openSUSE Leap 15.3.

#26 Updated by msmeissn 9 months ago

  • Subject changed from [qe-core] setup Leap Jump 15.3 openqa to [qe-core] setup openSUSE Leap 15.3 openqa

we need openqa setup, and it will be a bit different.

we can have incidents now for 2 different locations

  • openSUSE Backports SLE-15-SP3
    test incident: openSUSE:Maintenance:15678

  • openSUSE Leap 15.3

test incident: openSUSE:Maintenance:15681

both targets will go into Leap 15.3.

#27 Updated by tjyrinki_suse 9 months ago

  • Priority changed from Normal to Urgent

This needs bumping in priority and finding someone who can configure O3 accordingly.

#28 Updated by tjyrinki_suse 9 months ago

I've pinged osukup about this.

#29 Updated by szarate 8 months ago

  • Precedes action #76702: [qe-core][u] disable iso download in openQA added

#31 Updated by szarate 8 months ago

  • Description updated (diff)

#32 Updated by szarate 8 months ago

  • Description updated (diff)

Santiago to clarify if it's jump or leap

#33 Updated by szarate 8 months ago

  • Description updated (diff)

#34 Updated by msmeissn 8 months ago

it is called leap now. jump was only for 15.2.1, but 15.3 will be called leap, also in the directory paths.

#35 Updated by tjyrinki_suse 8 months ago

  • Description updated (diff)

#36 Updated by tjyrinki_suse 8 months ago

  • Description updated (diff)

#37 Updated by tjyrinki_suse 8 months ago

  • Due date deleted (2021-02-01)
  • Target version set to Ready

#38 Updated by tjyrinki_suse 8 months ago

  • Target version changed from Ready to QE-Core: Ready

#39 Updated by tjyrinki_suse 8 months ago

  • Target version changed from QE-Core: Ready to Ready

#40 Updated by szarate 8 months ago

  • Target version changed from Ready to QE-Core: Ready

#41 Updated by tjyrinki_suse 8 months ago

  • Description updated (diff)
  • Target version changed from QE-Core: Ready to future

#42 Updated by szarate 7 months ago

  • Assignee set to szarate
  • Target version changed from future to QE-Core: Ready

I'm picking this up while in training.

For now: https://github.com/openSUSE/openSUSE-release-tools/pull/2537 has been created based on the information that we have within the ticket.

#43 Updated by tjyrinki_suse 7 months ago

  • Status changed from Workable to In Progress

#44 Updated by szarate 7 months ago

  • How to test this?
  • Who consumes/what the json files modified in the PR
  • How do we check that the results are correct? (i.E that maintenance updates will be picked up)
  • What does the syncing of the jobgroups between Maintenance and OBS once the leap product is released

#45 Updated by msmeissn 7 months ago

someone needs to setup 15.3 job groups in openqa.opensuse.org. (antonios?)

There will be code changes needed in opensuse-release-tools and os-autoinst-distri-opensuse, as we now have 2 different looking incidents.

what is not handled yet, are incidents like this:

openSUSE:Maintenance:15953 hello

As we now have 2 sources for updates, openSUSE:Leap:15.3:Update AND openSUSE:Backports:SLE-15-SP3:Update to test for Leap 15.3.

openSUSE:Maintenance:15681 is the other possible case, with rpmlint-mini

BOTH cases must be handled.

#46 Updated by apappas 6 months ago

The code changes in the repos have been done, and the mediums and JobGrous have been set. Temporarily the beta iso is being used.

I expected that they woulkd have been triggered. I'll investigate why they haven't.

#47 Updated by apappas 6 months ago

The runs are now posted for "regular" updates but the bot continues to ignore the backports. I just asked the UM if it's possible to send a dummy update to the sle repo so that can be tested.

#48 Updated by msmeissn 6 months ago

openSUSE:Maintenance:15953 - recommended (low): hello
locked: 4
release: 879946, review, 39 days ago

maintenance_release: openSUSE:Maintenance:15953/hello.openSUSE_Backports_SLE-15-SP3_Update -> openSUSE:Backports:SLE-15-SP3:Update/hello.15953
maintenance_release: openSUSE:Maintenance:15953/patchinfo -> openSUSE:Backports:SLE-15-SP3:Update/patchinfo.15953

It was missing a qam-openqa review, I added one just now.

#49 Updated by apappas 6 months ago

  • Related to action #91827: [qe-core] Write tests and documentation for oqamaint bot added

#50 Updated by szarate 6 months ago

  • Assignee changed from szarate to apappas

I think for now, Anton has done more than I have :)

#51 Updated by okurz 5 months ago

  • Subject changed from [qe-core] setup openSUSE Leap 15.3 openqa to [qe-core] setup openSUSE Leap 15.3 openQA

Had a meeting with hrommel, vpelcak, msmeissn, mawerner about two aspects

  1. "repository setup in openQA" which is mainly covered in this ticket. Updates to openSUSE Leap 15.3 can come from 3 different projects:
    • SLE 15 SP3 shared packages
    • PackageHub 'Backports'
    • packages specific to openSUSE Leap 15.3

It is expected that QE-Core takes care of that. In best case this can be fixed with just configuration changes for the openSUSE Maintenance openQA bot, in worst case this might need more changes https://github.com/openSUSE/openSUSE-release-tools/blob/master/openqa-maintenance.py . msmeissn has reminded of the release date of Leap 15.3 in just a couple of days until this issue has been fixed

  1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

#52 Updated by apappas 5 months ago

The configuration for the packages specific to Leap 15.3 has been done and has been going on for some time. The problem is doing the rest of them. Complicating that, is, that to our knowledge is qe-core, the only documentation we have is the code, which means we just can't copy paste the config like we did before.

#53 Updated by tjyrinki_suse 5 months ago

okurz wrote:

It is expected that QE-Core takes care of that. In best case this can be fixed with just configuration changes for the openSUSE Maintenance openQA bot, in worst case this might need more changes https://github.com/openSUSE/openSUSE-release-tools/blob/master/openqa-maintenance.py . msmeissn has reminded of the release date of Leap 15.3 in just a couple of days until this issue has been fixed

Ok.

  1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

#54 Updated by okurz 5 months ago

tjyrinki_suse wrote:

  1. To avoid iterations after release of SLE 15 SP3 updates caused by failures that show only on openSUSE Leap 15.3, we should incorporate the review of openQA results of openSUSE Leap 15.3 into the release decision of SLE 15 SP3 updates. For this I suggested to extend the existing SLE maintenance tests on openqa.suse.de with according Leap test scenarios. Of course, a good start is to just add one scenario in either aggregate or incident tests and slowly extend from there. hrommel1 as discussed plans to create a specific ticket for that. Responsibility area for the initial setup is QE-Core as well.

If we don't look at O3 results (I think it would be possible to watch a subset of those results for the decision) but want to instead add Leap tests to OSD, then yes makes sense. Respective squads would be adding their tests, butfirst any media/product adding of Leap to OSD would be on QE Core.

The main and only reason to use openqa.suse.de is because the unreleased SLE maintenance updates are not available on openqa.opensuse.org. This should not be the default for new tests that run on Leap as Leap is still a community project which should be developed and tested in the open where possible.

#55 Updated by apappas 5 months ago

  • Status changed from In Progress to Feedback

The Backports part of the ticket has been completeted

What remains are the Leap SLE updates. According to Marcus, there is no Build Service project that publishes these updates, but there is a script that collects SLE15GA->SLE15SP# into the Leap SLE repo. He suggests we test by adding the SLE15SP3 testrepos to a Leap 15.3 installation.

I personally need feedback as to what the test scenarios for those updates will be.

#56 Updated by tjyrinki_suse 5 months ago

  • Parent task set to #92563

For SLE updates on Leap, I'm suggesting new job groups in OSD that test the same things as these job groups on O3, but by using the prospective SLE incident updates:

I'd suggest starting with cryptlvm/cryptlvm@uefi-2G, gnome/gnome@uefi, textmode@64bit and install_with_updates_gnome - and if those go fine, add KDE as well. x86_64 only! Requires creating also Leap products etc on OSD.

Heiko agreed this sounds like a good initial plan.

#57 Updated by okurz 5 months ago

how about starting with a single scenario as proof of concept? Especially as there would be no help from any openSUSE community to review and as people are already suffering from the review workload the additional tests should be really limited to only show that the SLE update side of things is not completely broken while keeping the main test load still on o3, right?

#58 Updated by tjyrinki_suse 5 months ago

  • Status changed from Feedback to In Progress

I think the above should be enough to get forward with the OSD SLE-on-Leap updates.

#59 Updated by maritawerner 3 months ago

Today there was a meeting with Marcus Meissner, Antonios, Santi, Anna and myself with this topics:

  • Anton has finished the set-up of the channels for 15.3, all good
  • For 15.4 Lubos will create a progress ticket at Beta1 to inform QE that openQa needs to be set-up for SP4
  • The review of the Leap maintenance job groups is still under discussion, Santi will follow-up with an email, the idea is that the QE Core team will take over the review -> more info will come
  • Anna & QE Core team will maintain the tests for Leap and increase the test coverage step by step

#60 Updated by okurz 3 months ago

waiting for apappas and osukup regarding scheduling tests within OSD. Would it better if we can not wait for 1:1 talks but better coordinate on team level. Maybe I can offer some help?

#61 Updated by okurz 3 months ago

There was no update and we are in the dark here. We should have apappas provide the actual requirements.

#62 Updated by apappas 3 months ago

  • Status changed from In Progress to Blocked

I can't get the openqabot to post the jobs for the job group I made. I tried with the help of geor, mgriffalconi and phoenix to fix my MR but we couldn't. I am setting this to blocked till some help arrives.

#63 Updated by okurz 3 months ago

  • Status changed from Blocked to Feedback

Please stick to https://progress.opensuse.org/projects/openqav3/wiki/Wiki#ticket-workflow and use "Blocked" only when there is some external dependency (e.g. bugs, not implemented features). There should be a link to another ticket, bug, trello card, etc. where it can be seen what the ticket is blocked by. You can use "Feedback" though: "Further work on the ticket is blocked by open points or is awaiting for the feedback to proceed. Sometimes also used to ask Assignee about progress on inactivity."

#64 Updated by maritawerner 3 months ago

Thanks for the update, Oli. But could you or somebody else please specify the "help" that Anton is waiting for? Is it the tools team, is it Ondrey Sukup or who else? On Wednesday in the call I had the impression that we are waiting for Anton. He has said that he can not solve the problem so we are going in cycles here.

#65 Updated by apappas 3 months ago

I am waiting for Ondrej or people that have knowledge of the openqa maintenance bot to help me edit the ΜR(1) so that it can post the jobs.

  1. https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/503

#67 Updated by szarate 3 months ago

apappas wrote:

I am waiting for Ondrej or people that have knowledge of the openqa maintenance bot to help me edit the ΜR(1) so that it can post the jobs.

  1. https://gitlab.suse.de/qa-maintenance/metadata/-/merge_requests/503

Ondrej can you help to move this forward?

#68 Updated by okurz 3 months ago

We discussed this in the weekly SUSE QE Sync 2021-08-04. Responsibility for qa-maintenance/openQABot itself is for sure within the scope of SUSE QE Tools. But enabling "openSUSE repo tests as part of SLE updates" is certainly a new domain and not something we have experience with. Just don't expect too much :)

#69 Updated by apappas 3 months ago

I am not aware of the current limits of the bot, but I want to make it clear that what is needed is, to take incidents from the SUSE:Updates:openSUSE-SLE:15.3 internal build service project and post the corresponding jobs for an already set up OSD JobGroup. AFAICT this is already what it does for the SLE projects.

#70 Updated by osukup 2 months ago

apappas wrote:

I am not aware of the current limits of the bot, but I want to make it clear that what is needed is, to take incidents from the SUSE:Updates:openSUSE-SLE:15.3 internal build service project and post the corresponding jobs for an already set up OSD JobGroup. AFAICT this is already what it does for the SLE projects.

OK, any example of incident with populated openSUSE-SLE:15.3 ?

  • we now doing switch to shedule based on qem-dashboard with different format of config files .. can be found in qa-maintenance/metadata/bot-ng

#71 Updated by msmeissn 2 months ago

SUSE:Maintenance:20230:245273
e.g. is a dracut update with openSUSE-SLE-15.3 in it.

https://smelt.suse.de/overview/ -> Testing -> search for openSUSE in the products.

#72 Updated by osukup 2 months ago

msmeissn wrote:

SUSE:Maintenance:20230:245273
e.g. is a dracut update with openSUSE-SLE-15.3 in it.

https://smelt.suse.de/overview/ -> Testing -> search for openSUSE in the products.

thx

openSUSE-SLE has a different repo structure in IBS so it needs add few workaround into bot:(

#73 Updated by osukup 2 months ago

  • Related to action #96752: 'openSUSE-SLE' product schedule jobs to OSD added

#75 Updated by osukup 2 months ago

jobs are scheduled in osd, but there is a missing Leap 15.3 ISO and the second question is if Leap needles are deployed for OSD jobs

#76 Updated by okurz about 2 months ago

Discussed in weekly QE sync 2021-09-01: This ticket was mentioned again by multiple stakeholders and should be worked on as soon as apappas is available again. apappas if you need support from members of SUSE QE Tools please reach out to us. So far I see the prerequisites in place. Of course test scheduling details need to be ruled out.

#77 Updated by szarate about 1 month ago

Waiting for Anton to come back next monday

#78 Updated by apappas about 1 month ago

  • Status changed from Feedback to In Progress

I have rescheduled the tests that were disabled, while I was away. It probably won't take much to make them fail less spectacularly. I will pull the teams in for tests where qe-core is not responsible, eg. cryptlvm

#79 Updated by apappas about 1 month ago

  • Status changed from In Progress to Feedback

IMHO the group is setup, the scenarios need some love by the various teams (needles etc.) and the schedule needs defining but I think that's a topic for another ticket. I am asking for feedback to resolve this ticket and make new ones.

#80 Updated by okurz about 1 month ago

apappas seems like you enabled broken Leap tests in production job groups so we have scenarios that "never worked" affecting releases of updates. Please do not do something like this. For example there is https://openqa.suse.de/tests/7142084#next_previous

Can you revert this ASAP and/or move tests to the "development" job group to first stabilize before moving into the production job groups?

EDIT: Wait, they are in the development job group like https://openqa.suse.de/tests/overview?distri=opensuse&version=15.3&build=%3A21237%3Asamba&groupid=393 . I think I accused you wrongly, sorry.

#81 Updated by okurz about 1 month ago

apappas wrote:

IMHO the group is setup, the scenarios need some love by the various teams (needles etc.) and the schedule needs defining but I think that's a topic for another ticket. I am asking for feedback to resolve this ticket and make new ones.

We discussed this in weekly QE sync 2021-09-22 and clarified: apappas yes, please create new tickets (first!) assigned to the according squads as further subtasks of #92563 and then (afterwards!) resolve this ticket.

#82 Updated by apappas 24 days ago

  • Category deleted (New test)
  • Status changed from Feedback to Resolved
  • Assignee deleted (apappas)
  • Priority changed from Urgent to Low

#84 Updated by pvorel 21 days ago

I wonder if it's desired to have also kernel testing for Leap 15.3. Just out of curiosity, because kernel QE team is that busy, that even Tumbleweed is not reviewed and tagged regularly (I try to watch it, but SLE and other projects are the priority).

#85 Updated by okurz 21 days ago

pvorel wrote:

I wonder if it's desired to have also kernel testing for Leap 15.3. Just out of curiosity, because kernel QE team is that busy, that even Tumbleweed is not reviewed and tagged regularly (I try to watch it, but SLE and other projects are the priority).

I consider the risk of breaking Leap with SLE kernel updates low considering that with Leap 15.3 after the "Jump" project the very same SLE binaries are used in Leap. Implicitly of course the kernel is tested against Leap with the existing Leap (for SLE maintenance updates) tests. As you said you are too busy so better focus on Tumbleweed tests with the more recent kernel versions before considering extending Leap kernel specific tests.

#86 Updated by msmeissn 21 days ago

the kernel opensuse leap 15.3 uses is tested by the SLE kernel testing... so I think it does not need it seperately for openSUSE at this time.

Also available in: Atom PDF