action #135731
closedaction #137189: [core][opensuse] Enable Slowroll tests
[tools][opensuse] Enable Slowroll tests size:M
0%
Description
Motivation¶
Related to ALP development as well as the future of openSUSE Leap a new distribution "Slowroll" is being setup. With the power of openQA we can ensure proper quality with very efficient human resources involved. This can also bring along good marketing
Acceptance criteria¶
- AC1: Every new build of Slowroll triggers at least one production test scenario (subset of Tumbleweed)
- AC2: No obvious infrastructure related setup problems apparent in tests
Suggestions¶
- Read https://en.opensuse.org/openSUSE:Slowroll
- Try out single openQA jobs, e.g. with
openqa-clone-job --within-instance https://openqa.opensuse.org/t$id_of_some_default_tumbleweed_test $needed_variables ISO_URL=$slowroll_iso_url - Extend https://github.com/os-autoinst/openqa-trigger-from-obs/ based on openSUSE:Factory / Tumbleweed
- Coordinate with bmwiedemann necessary further steps
- Try with the "mediacheck" (could be the fastest) or "default" or "textmode" scenario as a start, consider filing follow-up tickets if enabling tests becomes more involved
- Support bmwiedemann or others to extend from a single scenario, don't do all on yourself, just support from here :)
Out of scope¶
- Setting up all tests on our own
- Staging tests
Updated by okurz 7 months ago
- Copied to action #137189: [core][opensuse] Enable Slowroll tests added
Updated by mkittler 5 months ago
Judging by the instructions on https://en.opensuse.org/openSUSE:Slowroll#Use the setup still requires disabling online repositories during the installation and manual changes to repositories. Do we really want to put effort into automating that (considering these manual steps hopefully won't be required for long anymore)?
Updated by ybonatakis 4 months ago
I downloaded the DVD iso from https://download.opensuse.org/slowroll/iso/ into o3(fixed/). Most of the first basics test cases seems ok.
One test which failed was https://openqa.opensuse.org/tests/3849913#step/updates_packagekit_gpk/3 more due to the needling.
Note that the tests above are without the proper repos as they are described on https://en.opensuse.org/openSUSE:Slowroll
For this i guess, we need to update some tests modules or something.
Updated by ybonatakis 4 months ago
- Status changed from Workable to In Progress
product with the following variables created:
DISTRI: opensuse
VERSION: Slowroll
FLAVOR: DVD
Jobgroup is also created[0]
And enable 3 test cases.
We can test only x86_64 at the moment.
Tests however fail with the current setup.
The goal is to test update and staging repos at the moment.
Updated by ybonatakis 4 months ago
at least mediacheck test now works after some needling.
For the rest i created https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/806 to fix the operations when the DISABLE_ONLINE_REPOS is used. Based on the Bernhard Slowroll needs the DVD in offline mode. And as such i enabled DISABLE_ONLINE_REPOS in Slowroll test variables.
Once this is merged, some modification will need to add the Slowroll repos
Updated by openqa_review 4 months ago
- Due date set to 2024-01-24
Setting due date based on mean cycle time of SUSE QE Tools
Updated by ybonatakis 4 months ago
- Status changed from In Progress to Feedback
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18422 created which adds the Slowroll repos. https://aquarius.suse.cz/tests/20021#step/slowroll_repos/1
I am putting it on Feedback. I dont think the requirements are fulfilled. The jobgroup is added in Development Group and i believe this is not triggered automatically. I will talk with the QA-Core for followup steps
Updated by okurz 4 months ago
ybonatakis wrote in #note-14:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18422 created which adds the Slowroll repos. https://aquarius.suse.cz/tests/20021#step/slowroll_repos/1
I am putting it on Feedback. I dont think the requirements are fulfilled. The jobgroup is added in Development Group and i believe this is not triggered automatically. I will talk with the QA-Core for followup steps
please see https://progress.opensuse.org/issues/135731#Suggestions which mention https://github.com/os-autoinst/openqa-trigger-from-obs/ for automatic triggering. Also https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18422 adds Slowroll repos for the released product. I don't see how that makes sense. I suggest to compare against Tumbleweed Staging and development tests how they look like.
Updated by livdywan 3 months ago
Also https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18422 adds Slowroll repos for the released product
Please check the open questions. I see at least one q regarding the maintainer - a brief comment is enough, or a reminder in Slack if you were waiting on someone e.g. @szarate to get back to you
And the next step as discussed would be Extend https://github.com/os-autoinst/openqa-trigger-from-obs/ based on openSUSE:Factory / Tumbleweed
Updated by ybonatakis 3 months ago
- Due date deleted (
2024-01-24) - Status changed from In Progress to Workable
I tried to follow the steps from https://github.com/os-autoinst/openqa-trigger-from-obs/?tab=readme-ov-file#adding-new-projects
Step4 has to run inside ariel.
Able to login with ssh keys but once i run the rsync command it asks me to authenticate on obspublish(10.151.135.4) host.
no idea whats the password is. Stack at this point.
the rsync should generate the file_iso.lst and in theory I should be able to take it and continue local on my laptop.
Updated by openqa_review 3 months ago
- Due date set to 2024-02-01
Setting due date based on mean cycle time of SUSE QE Tools
Updated by ybonatakis 3 months ago
- Due date deleted (
2024-02-01)
ybonatakis wrote in #note-18:
I tried to follow the steps from https://github.com/os-autoinst/openqa-trigger-from-obs/?tab=readme-ov-file#adding-new-projects
Step4 has to run insideariel.
Able to login with ssh keys but once i run the rsync command it asks me to authenticate onobspublish(10.151.135.4) host.
So password can be found under /opt/openqa-trigger-from-obs/
However the command fails
rsync: [sender] change_dir "/openSUSE:Slowroll:Staging/images/x86_64/*product*" (in openqa) failed: No such file or directory (2)
no idea whats the password is. Stack at this point.
the rsync should generate the file_iso.lst and in theory I should be able to take it and continue local on my laptop.
Everything i did, is probably wrong.
there is no .iso https://build.opensuse.org/project/monitor/openSUSE:Slowroll:Staging?arch_x86_64=1&defaults=0&repo_images=1&succeeded=1
Full convo with Andrii who tried to help me out https://suse.slack.com/archives/C02CANHLANP/p1705501149107419
Updated by okurz 3 months ago
- Due date set to 2024-02-01
Well, regardless of your steps regarding testing out on ariel directly I suggest to create a draft PR in https://github.com/os-autoinst/openqa-trigger-from-obs/ and point more people to that draft PR like DimStar as Tumbleweed release manager who can help, Bernhard Wiedemann as RM for Slowroll and anikitin as openqa-trigger-from-obs domain expert
Updated by okurz 3 months ago
- Status changed from In Progress to Feedback
We discussed about this in the daily. I wrote a message in
https://suse.slack.com/archives/C05SPDVDRUH/p1705571207948599?thread_ts=1699372931.132459&cid=C05SPDVDRUH
So after talking to @Ioannis Bonatakis I understand that there are no iso files published in https://download.opensuse.org/repositories/openSUSE:/Slowroll:/Staging/images/?P=*.iso so openQA can not trigger any staging openQA tests. So, can isos be built and published for that project?
I suggest we wait for some answers in there.
Updated by okurz 3 months ago
- Description updated (diff)
- Status changed from Feedback to In Progress
(Oliver Kurz) Still no iso showing up in https://download.opensuse.org/repositories/openSUSE:/Slowroll:/Staging/images/?P=*.iso . But maybe those assets shouldn't actually be published on download.opensuse.org but just over "obspublish-stage" same as for Factory
But actually the ticket was never about staging tests. I added an explicit "out of scope" note now. Please focus on the original goals, i.e. that new snapshot builds are tested.
Updated by ybonatakis 3 months ago
- Status changed from In Progress to Workable
- Assignee deleted (
ybonatakis)
unassign myself as i am working on other things at the moment. gonna take a look when i will done with them
Updated by okurz 3 months ago
- Due date deleted (
2024-02-01) - Status changed from Workable to New
We decided to put this ticket back for re-estimation after it was clarified that some steps had been done for enabling "staging" tests when instead we were looking for production tests. I added an according out-of-scope section. Maybe in estimation you can just agree with the current description content and give it an estimate again.
Updated by openqa_review 3 months ago
- Due date set to 2024-02-14
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan 3 months ago
Discussed in the Unlock. Relevant notes:
- https://github.com/os-autoinst/openqa-trigger-from-obs has some documentation but it's omitting crucial parts like how it interacts with the ObsRsync plugin for openQA
- We need a new XML file, and a new job group (there is one but only for staging - remember we don't care about staging for now)
- https://github.com/os-autoinst/openqa-trigger-from-obs/commit/f16bc25a10d606afa0c070e5d0a9cabb556dbb88
- https://openqa.opensuse.org/tests/3905404
- Check with Bernhard or Fabian, they've added new XML configs most recently
- Let us know what comes out of it so we can reduce the bus factor
Updated by livdywan 3 months ago
As mentioned in the daily, using https://github.com/os-autoinst/openqa-trigger-from-obs/pull/239 is likely the best option now.
Updated by dheidler 3 months ago · Edited
The pr looks good so far and seems to produce working scripts.
What I found out so far that needs to be done to get a project tested in openQA:
- add xml xml/obs/openSUSE:Slowroll:Staging-images.xml
- on ariel:
- cd /opt/openqa-trigger-from-obs
- git pull --rebase
- run make openSUSE:Slowroll:Staging
- cd openSUSE:Slowroll:Staging
on this folder there have some scripts been generated for syncing images and tirggering jobs
openQA will call
/opt/openqa-trigger-from-obs/script/rsync.sh $projectwhich will call those scripts when api route /obs_rsync/#project/runs is calledthat api route seems to be called by https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/notify-obs_rsync.py
notify-obs_rsync.py gets the list of projects to handle from https://openqa.opensuse.org/api/v1/obs_rsync (openqa-cli api --o3 obs_rsync)
which will just look at the subdirs of /opt/openqa-trigger-from-obs that do NOT match^(t|profiles|script|xml)$
Updated by livdywan 3 months ago · Edited
So how do we find out if this works? Things look a little inconsistent:
- In the product log the most recent Slowroll I see is 23 days old
- The OBS Sync dashboard says
published on 2024-02-07 11:01:14 +0000 - The latest Slowroll build openQA knows about is 29 days old
- Minion dashboard ran a
obs_rsync_runjob 2 hours ago with no errors- On OBS it says standard is building, images is published at the time of this writing (this is not a permalink)
Also: Do we care about build errors on OBS?
Updated by livdywan 3 months ago
- Status changed from Resolved to Feedback
dheidler wrote in #note-38:
ACs are fulfilled so far.
I talked to Bernhard about the next steps.
How are the AC's fulfilled? I can find one build and one job in the last month. Please only resolve it after demonstrating that new builds have Slowroll tests, with openQA or OBS or whatever relevant instances.
If you are blocked i.e. there is an issue with builds you can also block on another ticket. Otherwise feedback it is.
Updated by okurz 2 months ago
- Assignee changed from dheidler to okurz
- Target version changed from Ready to Tools - Next
As discussed with livdywan will take over to act as point of contact for stakeholders, e.g. bmwiedemann. Because this ticket turned out costly and the currently written goals explicitly excluding staging but the recent work was only about staging we would need to re-evaluate before investing anymore significant effort.
Updated by szarate 2 months ago
- Parent task set to #137189
okurz wrote in #note-41:
As discussed with livdywan will take over to act as point of contact for stakeholders, e.g. bmwiedemann. Because this ticket turned out costly and the currently written goals explicitly excluding staging but the recent work was only about staging we would need to re-evaluate before investing anymore significant effort.
While its great to have staging, I was under the impression that most of the work needed was only syncing assets and using the work we have for leap. And in terms of Staging there are also some considerations that have to be made, #129931 mostly commenting here atm to link things.
let me know if there's something needed from Core, because while I see the tag of reactive work, at least for Core, it is planned work.
Updated by dheidler 2 months ago
Bernhard wrote today:
https://download.opensuse.org/repositories/openSUSE:/Slowroll:/Staging/images/ got a new openSUSE-Tumbleweed-Minimal-VM.x86_64-1.0.0-kvm-and-xen-Build4.25.qcow2 but openqa doesn't triggert on it
Let's see if this helps:
https://github.com/openSUSE/openSUSE-release-tools/pull/3060
Updated by okurz about 1 month ago
https://github.com/openSUSE/openSUSE-release-tools/pull/3060 merged since long and as far as I read the minutes from openSUSE release meetings Slowroll is fine with just staging openQA tests so far.
@bmwiedemann https://en.opensuse.org/openSUSE:Slowroll still states "Untested packages go into https://build.opensuse.org/project/show/openSUSE:Slowroll:Staging first and are tested by openQA (TBD)". What needs to be done so that we can state that Slowroll is sufficiently tested by openQA?
Updated by bmwiedemann about 1 month ago
There are some workflows that still need re-design. It is not only related to openQA.
Currently we only test one JeOS image in openQA, while the documented way of installation is to use the old Tumbleweed DVD iso snapshot in an offline install and then change the repos.
That install process still needs to be improved. https://build.opensuse.org/request/show/1162881 can help a bit there.
We could also test upgrades from Leap 15.5 to Slowroll.
The JeOS qcow2 is built in the Staging context with package versions that might have been rebuilt after being released into openSUSE:Slowroll. So there is some risk that what we test is not what will be released.
Updated by okurz about 1 month ago
ok, all those suggestions sound sensible to me but guess you don't need exactly the QE Tools team to do that then, right? So should we declare this ticket with the original goal to have "some" production tests as "Resolved"?