QA » openQA Project

There were two rounds of product triggering:

Scheduled product 1980120

Scheduled product 1980232

I raised the topic of multiple openqa triggering of same build , in https://suse.slack.com/archives/C02CANHLANP/p1698981583137759. Hope more attention can be given and avoid it.

@ybonatakis @emiura I am not sure if I am reaching the same ones that have openqa accounts mentioned in https://progress.opensuse.org/issues/138593#note-7. Please forgive me if not. If it is correct, I would appreciate if you can share why you need to rerun such a large range of tests, including virtualization, for build 28.1. Thanks!

@szarate Hello Santi, would you please help a look why two times of geekotest triggering for the build? Any bug to fix on CI tool?

xlai wrote in #note-8:

I raised the topic of multiple openqa triggering of same build , in https://suse.slack.com/archives/C02CANHLANP/p1698981583137759. Hope more attention can be given and avoid it.

@ybonatakis @emiura I am not sure if I am reaching the same ones that have openqa accounts mentioned in https://progress.opensuse.org/issues/138593#note-7. Please forgive me if not. If it is correct, I would appreciate if you can share why you need to rerun such a large range of tests, including virtualization, for build 28.1. Thanks!

This meant to run only HPC test group from my side. Usually it just do that. I dont know why Virtualization tests re-triggered.

@szarate Hello Santi, would you please help a look why two times of geekotest triggering for the build? Any bug to fix on CI tool?

ybonatakis wrote in #note-9:

xlai wrote in #note-8:

I raised the topic of multiple openqa triggering of same build , in https://suse.slack.com/archives/C02CANHLANP/p1698981583137759. Hope more attention can be given and avoid it.

@ybonatakis @emiura I am not sure if I am reaching the same ones that have openqa accounts mentioned in https://progress.opensuse.org/issues/138593#note-7. Please forgive me if not. If it is correct, I would appreciate if you can share why you need to rerun such a large range of tests, including virtualization, for build 28.1. Thanks!

This meant to run only HPC test group from my side. Usually it just do that. I dont know why Virtualization tests re-triggered.

@ybonatakis thanks for the reply. Would you please paste your trigger command here, so that the relevant team may help to find out the reason? From https://openqa.suse.de/admin/productlog?id=1988393, there is a long successful job list which seems much more than HPC only.

BTW, did you specify "TEST=" in your command? If nothing given, but only mediums info provided, all tests binded to that would be rerun. That used to happen on OSD before by accident.

xlai wrote in #note-8:

I raised the topic of multiple openqa triggering of same build , in https://suse.slack.com/archives/C02CANHLANP/p1698981583137759. Hope more attention can be given and avoid it.

@ybonatakis @emiura I am not sure if I am reaching the same ones that have openqa accounts mentioned in https://progress.opensuse.org/issues/138593#note-7. Please forgive me if not. If it is correct, I would appreciate if you can share why you need to rerun such a large range of tests, including virtualization, for build 28.1. Thanks!

@szarate Hello Santi, would you please help a look why two times of geekotest triggering for the build? Any bug to fix on CI tool?

@xlai
I'm sorry for the trouble, my intention was to restart the teststo check if registry related errors had been fixed. It will not happen again.

emiura wrote in #note-11:

@xlai
I'm sorry for the trouble, my intention was to restart the teststo check if registry related errors had been fixed. It will not happen again.

No worries. Thank you! @emiura

For Build34.1 and Build37.1, the historical scheduled products are as below:

1992810 a day ago   epaolantonio    scheduled   SLE 15-SP6  Online  x86_64  37.1    SLE-15-SP6-Online-x86_64-Build37.1-Media1.iso   
1990407 8 days ago  geekotest   scheduled   SLE 15-SP6  Online  x86_64  34.1    SLE-15-SP6-Online-x86_64-Build34.1-Media1.iso   

@ybonatakis It seems you scheduled test for Build39.1 again. All virtualization tests are running again now.

about an hour ago ybonatakis scheduled sle 15-SP6 Online x86_64 39.1 SLE-15-SP6-Online-x86_64-Build39.1-Media1.iso

Why do you need to do this ?

Similar thing happens again. Individuals triggers full build test accidentally. It's also raised and discussed in https://suse.slack.com/archives/C0564EU665S/p1709176558631769.

Just for clarification:
The historical comments and the original ticket description may mix two issues up.
- In this ticket, please focus on what the ticket title gives -- Restart of scheduled products is prone to retriggers by humans, which will need the help from openqa tool to improve. For anyone helps with it, please focus on related comments for it only.
- the other issue of multiple triggers by geekotest has been scoped into a new ticket #156277.

I am new to the discussion but as a user who have fell into the mistake of trigger incidentally jobs for other job groups I want to say that I disagree with the suggestion of an admin permission for that. Usually the problem is the misuse or misunderstanding of pressing a button or of openqa-cli. Instead, I would like to have a prompt and a warning before the execution.
For example when i do an post isos from the console, and if this has a results to schedule a whole job group, I would expect a prompt which shows me what it is going to schedule and ask for confirmation to continue.

I also question both the ACs

• "RBAC is implemented or an Epic for its implementation exists": IMHO that already exists. We already have three different levels with user, operator, admin. What do you see as missing here?
• Restart of Scheduled produtcs is put behind a higher level permission (Admin?) Same as ybonatakis stated that wouldn't help: we have many admins and they can still make a mistake. And others might still have a valid need to retrigger. I would like us to check why we even have that retrigger button on the job details page as we already have one in the "scheduled products" page where I think it shows more clearly that many jobs would be affected

The entire build 59.2 was retriggered again ...

about 11 hours ago  coolo   scheduled   SLE 15-SP6  Online  x86_64  59.2    SLE-15-SP6-Online-x86_64-Build59.2-Media1.iso

One more retrigger ...

about 12 hours ago  dheidler    scheduled   SLE 15-SP6  Online  x86_64  59.2    SLE-15-SP6-Online-x86_64-Build59.2-Media1.iso

okurz wrote in #note-24:

No response on #138593-21, lowering prio

I would like us to check why we even have that retrigger button on the job details page as we already have one in the "scheduled products" page where I think it shows more clearly that many jobs would be affected

I vote for this too. Maybe removing the button of rescheduling products from job details page can decrease such accidental trigger. Anyone would like to do so, better to go to "scheduled products" page.

Then how about those accidental triggers from command line via openqa-cli?

one more time again today:

about 4 hours ago   llzhao  scheduled   SLE 15-SP6  Online  x86_64  88.1    SLE-15-SP6-Online-x86_64-Build88.1-Media1.iso

okurz wrote in #note-31:

Careful when changing the fields. This ticket is not workable according to https://progress.opensuse.org/projects/qa/wiki/Tools#Definition-of-READY-for-new-features

Oops, sorry. Which item do you think is missing for workable?

We will look into this with high priority. Be aware though that we don't have sprints.

I see.

Removing the button might be easy but also it was added based on a certain request so it's not as easy as that. But I am certain we will be able to find a better solution

Cool, thanks!
BTW, just for your information, I learned that llzhao reran the whole product, because she did not find a way to rerun only her job group. Maybe this can be considered within this feature request or a new one.

xlai wrote in #note-32:

okurz wrote in #note-31:

Careful when changing the fields. This ticket is not workable according to https://progress.opensuse.org/projects/qa/wiki/Tools#Definition-of-READY-for-new-features

Oops, sorry. Which item do you think is missing for workable?

The ticket has good details but we need the tools team to estimate the ticket and verify that we have all necessary information about the goal and starting steps

BTW, just for your information, I learned that llzhao reran the whole product, because she did not find a way to rerun only her job group. Maybe this can be considered within this feature request or a new one.

That would be related to
#157342 but "just one job group" is a kinda special request. We will have to see if it fits within what we can do with the webUI or just CLI tooling

The PR has been merged and deployed (on o3 at least). I got a "thumbs-up" reaction on the PR by @Julie_CAO.

The feedback about the buttons I got on Slack is also good. The explicitness also triggered some questions - but I guess that's a good thing. Some minor tweaks to apply (considering the Slack conversation so far):

• Don't show "Actions:" at all if there are no actions to show. It already doesn't show if the user has not the right permissions but it shows on jobs that have already been restarted. Maybe it would make more sense to make that explicit (instead of hiding the button completely), e.g. show the restart button but disabled and with a popover stating that the job has already a clone so it cannot be restarted.
• Maybe mention in the help popover for restating where to find the advanced options. (The "<" on the left is maybe a bit too subtle on its own.)

Hi @mkittler ,
Thanks for the effort on this.

To me, it is much more differentiated for the two rerun buttons now. I believe it will give less chance that the rerun button is clicked by mistake. Nice improvement!

I still have some confusion here on the explanation for the button Reschedule product from here in job details page. Would you please explain the difference between it and rescheduling product tests in https://openqa.suse.de/admin/productlog?

@xlai Note that the "Reschedule from here" button was added for #124469. It is not just a shortcut for the button on the product log page. The requirement was to re-schedule a product only partially starting from a specific job. Only this "start job" and its child dependencies (but not parents) were supposed to be scheduled again. So that is what this feature does - as explained in the help text/icon next to the button.

I'll see whether it makes sense to implement ideas from #138593#note-39 after the bootstrap migration (because it would cause conflicts).

What was the openqa-cli command that was invoked accidentally? And what was the command that should have been invoked instead? With that information we could think how to improve openqa-cli. However, we should keep in mind that breaking compatibility of openqa-cli is probably a bad idea as well. Maybe we can create a new user friendly sub command for whatever the use case was or add a --dry-run feature to the existing command and advertise its use.

mkittler wrote in #note-48:

What was the openqa-cli command that was invoked accidentally? And what was the command that should have been invoked instead? With that information we could think how to improve openqa-cli. However, we should keep in mind that breaking compatibility of openqa-cli is probably a bad idea as well. Maybe we can create a new user friendly sub command for whatever the use case was or add a --dry-run feature to the existing command and advertise its use.

@ybonatakis Would you please provide some input for above based on your #note-20?

ybonatakis wrote in #note-20:

I am new to the discussion but as a user who have fell into the mistake of trigger incidentally jobs for other job groups I want to say that I disagree with the suggestion of an admin permission for that. Usually the problem is the misuse or misunderstanding of pressing a button or of openqa-cli. Instead, I would like to have a prompt and a warning before the execution.
For example when i do an post isos from the console, and if this has a results to schedule a whole job group, I would expect a prompt which shows me what it is going to schedule and ask for confirmation to continue.

@mkittler
This morning, there is one more problematic product rerun from job details page via Reschedule product from here button. To me, it seems a bug of the button that it does not only trigger CHILD jobs, but only jobs without dependency at all.

Rescheduled product link is https://openqa.suse.de/admin/productlog?id=2180274. From it, you can see overall 974 jobs got rerun.

The owner (@tinawang123 ) did not remember exactly the job that she clicked the button. But she confirmed below:

• she just clicked Reschedule product from here button from one of the failure jobs in Yast 15sp6 job group, https://openqa.suse.de/tests/overview?distri=sle&version=15-SP6&groupid=129&build=93.5.
• The most complex dependency is this , https://openqa.suse.de/tests/14611144#dependencies, less than 900. If only CHILD Jobs are supposed to be rerun, it shall be less than that.
• she guess the job that she clicks reschedule product button is yast2_cmd@64bit, the link given above, but I can't tell now because the wrong triggered jobs have been deleted by her
• I observed irrelevant jobs (not child or parent jobs to any of the job in the group) got rerun via this test, https://openqa.suse.de/tests/14610325#next_previous. But now the job has been deleted via openqa-cli. You shall be able to get it from logs.

More background info:

• why she clicks that button Reschedule product from here: she updated some settings, then click the Restart this and dependent jobs won't have the updated setting, so she clicked it

Maybe this is one of the main root causes for the initial problem -- virtualization jobs got rerun multiple times? Please help have a look. Thanks.

Ok, that's useful information. The number of re-triggered jobs should indeed not exceed 900 in that case. I will import a production database dumb and see whether I can reproduce the problem.

Note that she was also generally correct to click the button to make use of updated settings from tables like test suites. This aspect was one of the reasons why we initially added the feature. It might be helpful to know what settings have changed. If dependency-related settings have changed this might make a difference.

mkittler wrote in #note-51:

Ok, that's useful information. The number of re-triggered jobs should indeed not exceed 900 in that case. I will import a production database dumb and see whether I can reproduce the problem.

Glad to know that. Thank you!

Note that she was also generally correct to click the button to make use of updated settings from tables like test suites. This aspect was one of the reasons why we initially added the feature. It might be helpful to know what settings have changed. If dependency-related settings have changed this might make a difference.

I asked @tinawang123 , she said she only added this setting START_AFTER_TEST: 'create_hdd_textmode', no other setting change.

Ok, that's useful information. The number of re-triggered jobs should indeed not exceed 900 in that case. I will import a production database dumb and see whether I can reproduce the problem.

@mkittler Thanks for the fixes. May I ask for some data? With your above 2 fixes, how many jobs shall actually be rerun for the scene that the problem was triggered?

@xlai I can't directly answer your question but as of today https://github.com/os-autoinst/openQA/pull/5707 is deployed also on OSD. Changelog notification message in https://mailman.suse.de/mlarch/SuSE/openqa/2024/openqa.2024.06/msg00014.html . If you are using the different retriggering features maybe you will observe the change in behaviour and can verify that it's working better now.

Thank you all. I will follow the new deployed fixes for a while.