Project

General

Custom queries

Profile

Actions

action #128399

closed

[bot-ng] Scheduler uses metadata from previous day size:M

Added by dzedro over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Start date:
2023-04-28
Due date:
% Done:

0%

Estimated time:

Description

[bot-ng] Scheduler uses metadata from previous day

Motivation

The bot-ng scheduler could use latest metadata.
If it's a big problem to do, then we can just wait,
but it would be nice to don't have this delay, especially for Incidents.

$ rpm --query qam-metadata-openqabot
qam-metadata-openqabot-20230427-qam.1.1.noarch

Acceptance criteria

  • AC1: qam-metadata-openqabot is synchronously provided "quick enough" based on consumer requirements

Suggestions

  • Crosscheck with dzedro about requirements. Do we actually need to have it quicker or just "it would be nice"?
  • Check the current trigger: If it's time-based consider using event-based triggering and ensure the metadata is packaged and provided in a quicker way
  • Consider switching to using the git metadata version directly (in an efficient way) unless we strongly rely on package based features
Actions #1

Updated by jbaier_cz over 1 year ago

Metadata for the bot-ng pipeline is taken from the base image (qam-ci-leap:latest).
The base image is located in https://build.suse.de/package/show/QA:Maintenance/openSUSE-Leap-Container and should be rebuilded whenever metadata package (qam-metadata-openqabot) change.
Metadata package is located in https://build.suse.de/package/show/QA:Maintenance/qam-metadata and is rebuilded by the CI after merge (like https://gitlab.suse.de/qa-maintenance/metadata/-/jobs/1539286).

In theory, there should be no delay. Practically, it depends on the IBS build/publish time and Gitlab (docker executor) image cache.

Just for the record, this approach was chosen as preferred over zypper in qam-metadata-openqabot because of reliability issues with the state of the repository (and of course speed, it is not efficient to do zypper ref/in every 30 minutes in each pipeline run).

Actions #2

Updated by jbaier_cz over 1 year ago

Additionaly, https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1539447 already uses the new metadata package which was created like 80 minutes ago. It does not feel like a big delay to me.

Actions #3

Updated by okurz over 1 year ago

  • Project changed from openQA Project (public) to QA (public)
  • Category deleted (Feature requests)
  • Target version set to Ready
Actions #4

Updated by dzedro over 1 year ago

jbaier_cz wrote:

Additionaly, https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1539447 already uses the new metadata package which was created like 80 minutes ago. It does not feel like a big delay to me.

I didn't know how often is the package build, I thought it's like once per day.
Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?

Actions #5

Updated by jbaier_cz over 1 year ago

dzedro wrote:

I didn't know how often is the package build, I thought it's like once per day.

It's build on every change in the corresponding metadata repository.

Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?

It would be Gitlab, assuming you mean metadata repository. Then every bot pipeline would need to clone the repository and basically duplicate all steps from the package build (running some python scripts and copying results to the corresponding paths for the bot, if I recall correctly).

It's doable, but a step in an opposite direction in my opinion.

Actions #6

Updated by dzedro over 1 year ago

jbaier_cz wrote:

dzedro wrote:

I didn't know how often is the package build, I thought it's like once per day.

It's build on every change in the corresponding metadata repository.

Only thing I think of is can't scheduler use github directly instead of waiting for IBS and package ?

It would be Gitlab, assuming you mean metadata repository. Then every bot pipeline would need to clone the repository and basically duplicate all steps from the package build (running some python scripts and copying results to the corresponding paths for the bot, if I recall correctly).

It's doable, but a step in an opposite direction in my opinion.

Yes, building package is not required just having latest metadata.
It's fine if it stays as it is, I already know there is some delay, can be bigger if ibs is loaded.
Thank you.

Actions #7

Updated by okurz over 1 year ago

  • Subject changed from [bot-ng] Scheduler uses metadata from previous day to [bot-ng] Scheduler uses metadata from previous day size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #9

Updated by dzedro over 1 year ago

All is solved but today is scheduler still using qam-metadata-openqabot-20230531-qam.5.2.noarch ...

Actions #10

Updated by jbaier_cz over 1 year ago

  • Assignee set to jbaier_cz

dzedro wrote:

All is solved but today is scheduler still using qam-metadata-openqabot-20230531-qam.5.2.noarch ...

Sure, that's the most recent version published. Newer version cannot be published, because someone broke the QA:Maintenance repository setup in IBS. That is another issue which I can try to fix.

Actions #11

Updated by dzedro over 1 year ago

jbaier_cz wrote:

dzedro wrote:

All is solved but today is scheduler still using qam-metadata-openqabot-20230531-qam.5.2.noarch ...

Sure, that's the most recent version published. Newer version cannot be published, because someone broke the QA:Maintenance repository setup in IBS. That is another issue which I can try to fix.

I have seen the slack message, Thank you!

Actions #12

Updated by jbaier_cz over 1 year ago

Just for the record, the issue is caused by some problem in OBS/IBS, the buildops team is aware of the issue and is working on it. There is currently nothing we can do.

Actions

Also available in: Atom PDF