QA » openQA Project

Interestingly, all of these jobs actually exist. When retrying the Minion jobs they succeed.

The relevant jobs where only scheduled for one second until they've got cancelled again. The cancellation triggers the finalize job and in these cases the finalize job was executed without further delay. Might it be that the cancellation is done within the same transaction which created the job? If this transaction wasn't concluded when the Minion job is executed it would make sense that it doesn't "see" it yet.

By the way, this problem could theoretically also happen if a job really gets deleted after it has just been finished but before the Minion job had a chance to run. However, that seems rather unlikely to happen in practice.

I see only one failed job from 4 hours ago. Likely the job you've mentioned has already been deleted.

The job I see now failed with "Job 4967343 does not exist." but in this case the job really doesn't exist. Likely it has existed at some point but has been deleted.

1. We could simply treat this case as a success. However, if we would really pass invalid job IDs we would simply ignore that mistake. It would also not solve errors when the openQA job is deleted during the execution of the Minion job.
2. We could also abort any related Minion jobs before deleting an openQA job. The deletion of an openQA job (triggered via API or from the cleanup code) is already a little bit involved anyways (as we delete the file on the file system and delete screenshot links) so dealing with Minion jobs there shouldn't make it much worse. Of course a Minion job can have multiple related openQA jobs so we should only stop it if no other openQA job is related.

the second approach sounds like a clean solution to me :)

EDIT: Oh, but actually if you think the first approach is much easier then I would just go with that and wait for the problem that we ignore the mistake of invalid job IDs :)

PR for option 1.: https://github.com/os-autoinst/openQA/pull/3550

The problem with option 2. is that the Gru tables don't actually allow to find the related Minion jobs of an openQA job. They only allow to find the Gru job but as far as I see there's not relation from that to the underlying Minion job. I would need to extend the Gru code for that but actually it would make more sense to phase it out completely turning GruDependencies into MinionDependencies (see #77704). So for now I would just go for option 1.. Of course we'll still see failing jobs when the deletion happens while the finalize jobs is executed but that's rather unlikely. (It is generally unlikely to see failed finalize_job_results jobs and I guess the low probabilities multiply.) Merging the PR for option 1. also means we suppress problems within the creation of finalize jobs (when a wrong job ID is passed).

Note that option 2. would also involve having to stop a possibly running Minion job. That could be done via a broadcast.

In case your PR does not fix all the cases we see we can still exclude the specific problem situation from our monitoring and alerting.