action #128399
closed[bot-ng] Scheduler uses metadata from previous day size:M
0%
Description
[bot-ng] Scheduler uses metadata from previous day
Motivation¶
The bot-ng scheduler could use latest metadata.
If it's a big problem to do, then we can just wait,
but it would be nice to don't have this delay, especially for Incidents.
$ rpm --query qam-metadata-openqabot
qam-metadata-openqabot-20230427-qam.1.1.noarch
Acceptance criteria¶
- AC1: qam-metadata-openqabot is synchronously provided "quick enough" based on consumer requirements
Suggestions¶
- Crosscheck with dzedro about requirements. Do we actually need to have it quicker or just "it would be nice"?
- Check the current trigger: If it's time-based consider using event-based triggering and ensure the metadata is packaged and provided in a quicker way
- Consider switching to using the git metadata version directly (in an efficient way) unless we strongly rely on package based features
Updated by jbaier_cz over 1 year ago
Metadata for the bot-ng pipeline is taken from the base image (qam-ci-leap:latest).
The base image is located in https://build.suse.de/package/show/QA:Maintenance/openSUSE-Leap-Container and should be rebuilded whenever metadata package (qam-metadata-openqabot) change.
Metadata package is located in https://build.suse.de/package/show/QA:Maintenance/qam-metadata and is rebuilded by the CI after merge (like https://gitlab.suse.de/qa-maintenance/metadata/-/jobs/1539286).
In theory, there should be no delay. Practically, it depends on the IBS build/publish time and Gitlab (docker executor) image cache.
Just for the record, this approach was chosen as preferred over zypper in qam-metadata-openqabot
because of reliability issues with the state of the repository (and of course speed, it is not efficient to do zypper ref/in every 30 minutes in each pipeline run).
Updated by jbaier_cz over 1 year ago
Additionaly, https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1539447 already uses the new metadata package which was created like 80 minutes ago. It does not feel like a big delay to me.
Updated by okurz over 1 year ago
- Project changed from openQA Project (public) to QA (public)
- Category deleted (
Feature requests) - Target version set to Ready
Updated by dzedro over 1 year ago
jbaier_cz wrote:
Additionaly, https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1539447 already uses the new metadata package which was created like 80 minutes ago. It does not feel like a big delay to me.
I didn't know how often is the package build, I thought it's like once per day.
Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?
Updated by jbaier_cz over 1 year ago
dzedro wrote:
I didn't know how often is the package build, I thought it's like once per day.
It's build on every change in the corresponding metadata repository.
Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?
It would be Gitlab, assuming you mean metadata repository. Then every bot pipeline would need to clone the repository and basically duplicate all steps from the package build (running some python scripts and copying results to the corresponding paths for the bot, if I recall correctly).
It's doable, but a step in an opposite direction in my opinion.
Updated by dzedro over 1 year ago
jbaier_cz wrote:
dzedro wrote:
I didn't know how often is the package build, I thought it's like once per day.
It's build on every change in the corresponding metadata repository.
Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?
It would be Gitlab, assuming you mean metadata repository. Then every bot pipeline would need to clone the repository and basically duplicate all steps from the package build (running some python scripts and copying results to the corresponding paths for the bot, if I recall correctly).
It's doable, but a step in an opposite direction in my opinion.
Yes, building package is not required just having latest metadata.
It's fine if it stays as it is, I already know there is some delay, can be bigger if ibs is loaded.
Thank you.
Updated by okurz over 1 year ago
- Subject changed from [bot-ng] Scheduler uses metadata from previous day to [bot-ng] Scheduler uses metadata from previous day size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by osukup over 1 year ago
- Status changed from Workable to Resolved
all question should be solved by https://progress.opensuse.org/issues/128399#note-5 and https://progress.opensuse.org/issues/128399#note-1 https://progress.opensuse.org/issues/128399#note-2
Updated by dzedro over 1 year ago
All is solved but today is scheduler still using qam-metadata-openqabot-20230531-qam.5.2.noarch
...
Updated by jbaier_cz over 1 year ago
- Assignee set to jbaier_cz
dzedro wrote:
All is solved but today is scheduler still using
qam-metadata-openqabot-20230531-qam.5.2.noarch
...
Sure, that's the most recent version published. Newer version cannot be published, because someone broke the QA:Maintenance repository setup in IBS. That is another issue which I can try to fix.
Updated by dzedro over 1 year ago
jbaier_cz wrote:
dzedro wrote:
All is solved but today is scheduler still using
qam-metadata-openqabot-20230531-qam.5.2.noarch
...Sure, that's the most recent version published. Newer version cannot be published, because someone broke the QA:Maintenance repository setup in IBS. That is another issue which I can try to fix.
I have seen the slack message, Thank you!
Updated by jbaier_cz over 1 year ago
Just for the record, the issue is caused by some problem in OBS/IBS, the buildops team is aware of the issue and is working on it. There is currently nothing we can do.