action #123797
closedContinuous retriggers of GNOME:Next openQA tests
0%
Description
Observation¶
Dimstar reached out to me noticing that https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=Gnome-Live&machine=64bit-4G&test=gnome-live&version=43.2 is recurringly retriggered.
Impact¶
openqa.opensuse.org is wasting ressources although likely this will not have a worse effect than that.
Problem¶
This happens because http://jenkins.qa.suse.de/job/gnome_next-openqa always detects a change in content in the URL "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/?P=GNOME_Next*", see http://jenkins.qa.suse.de/job/gnome_next-openqa/urltriggerPollLog/. Likely we also can not use the "last-modified" date as curl --head "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/"
shows that MirrorCache does not seem to deliver that.
Suggestions¶
- DONE As workaround throttle the builds in jenkins -> #123797#note-1
- Try out different options for the URL change monitor
- Discuss with MirrorBrain owners about possible solutions
- Consider migrating the pipeline to another solution. E.g. http://jenkins.qa.suse.de/job/krypton-openqa/ was replaced by openqa-trigger-from-obs+botmaster.suse.de with https://github.com/os-autoinst/openqa-trigger-from-obs/pull/151 and https://github.com/openSUSE/openSUSE-release-tools/pull/2684
Updated by okurz over 1 year ago
- Description updated (diff)
I enabled "Throttle builds" in http://jenkins.qa.suse.de/job/gnome_next-openqa/configure with max. 6 builds/day while allowing user triggered builds to skip the rate limit
Updated by okurz over 1 year ago
- Status changed from New to In Progress
- Assignee set to okurz
Previously the "URLTrigger" monitored http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/?P=GNOME_Next* but this is now served by MirrorCache which has no "last-modified" timestamp but also the content always changes as the generated HTML page shows a "csrf-token" changing on each call. I found that by calling
diff <(curl -sS "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/?P=GNOME_Next*") <(curl -sS "http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/?P=GNOME_Next*")
which yields:
< <meta name="csrf-token" content="3a72819665c9adb750ad4d5e8054961c5b1c3efc" />
---
> <meta name="csrf-token" content="0b17cbf73c393a1baa4daee989158038caeafc7c" />
I switched the URLTrigger to inspect the URL content of http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/GNOME_Next.x86_64.iso.sha256 which is small enough to not be a problem to be recurringly downloaded. Let's see if this works.
And I created https://github.com/openSUSE/MirrorCache/issues/349
Updated by openqa_review over 1 year ago
- Due date set to 2023-02-14
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz over 1 year ago
- Status changed from In Progress to Feedback
The current build history in http://jenkins.qa.suse.de/job/gnome_next-openqa/ looks like we trigger as many as the build throttle allows. But checking for example the latest finished build http://jenkins.qa.suse.de/job/gnome_next-openqa/7830/ from Feb 3, 2023, 12:53:00Z, comparing to http://download.opensuse.org/repositories/GNOME:/Medias/images/iso/ the latest modified is 1320Z so for sure there had been changes today. Let's monitor for longer.
Updated by okurz over 1 year ago
- Due date deleted (
2023-02-14) - Status changed from Feedback to Resolved
It seems the process is working fine. We seem to have sometimes multiple builds per day of GNOME Next but maybe we don't need that. So I will just keep the build throttle in place.