QA (public) » openQA Project (public)

Again in:

• https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3598446
• https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3600358
• https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3600359

https://github.com/os-autoinst/openQA/pull/6101 the "monitor" command we rely on here needs to support skipping of "aborted" jobs. Which it turns out we already have a constant for.

livdywan wrote in #note-22:

Couldn't really come up with a mitigation since retrying the pipeline won't help if a job is obsoleted.

Why wouldn't that help? It would schedule new jobs.

tinita wrote in #note-23:

livdywan wrote in #note-22:

Couldn't really come up with a mitigation since retrying the pipeline won't help if a job is obsoleted.

Why wouldn't that help? It would schedule new jobs.

It won't help ensuring the original jobs are known to pass or be superseded as mentioned in the GitHub pull request thread? But that is something I also wanted to discuss again to make sure we are on the same page.

livdywan wrote in #note-25:

It won't help ensuring the original jobs are known to pass or be superseded as mentioned in the GitHub pull request thread? But that is something I also wanted to discuss again to make sure we are on the same page.

My approach in the PR was: check if the job was obsoleted/cancelled and do the openqa-cli schedule call again.

1. add API query parameter "follow=1" to follow restarts
2. investigate why jobs ended up as "obsolete" as we don't trigger with obsoletion and no obsolete should be default -> crosscheck the builds are not obsoleted by default

I don't want to put it to Workable or Feedback right now, but investigation is going slower because of other on-going tickets.

• Pipelines are running but manually monitored (by me) without alert emails.
• No fix per se identified so far.

We are in fact explicitly excluding parallel children when cancelling jobs from the job cluster by default. This is how the code is written and it looks intentional but wrong at the same time. It explains the timeouted job we saw.

EDIT: I misread the not so straight forward code. So we actually do cancel the cluster completely by default. There must nevertheless be a bug in the logic.

EDIT: I couldn't find any problem with the code, though. I extended unit tests and the tests didn't show this problem: https://github.com/os-autoinst/openQA/pull/6130

EDIT: I'm not sure how the the skipped job even ended up skipped. There's nothing in the audit log.

I suppose at this point it isn't worth looking into this case any further.

So for now I'm just blocking this ticket on #175299.

The failure from today (https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3688628#L56) is actually a different issue, see #170209#note-38.

I checked my inbox again and saw that the pipeline also failed yesterday, and that failure was actually one of the cases we also saw previously when creating this ticket (combination of timeout exceeded + skipped, see https://openqa.suse.de/tests/16512096 and https://openqa.suse.de/tests/16512097). So maybe worth looking into a little bit further after all.

The pipelines are also still running. Depending on what I can find out (or fix) I'll see whether it makes sense to pause them again.

I think the suggestion

Check whether we cancel the full parallel cluster in case a job in it is cancelled/obsoleted as we also saw jobs with parallel dependencies ending up with the result "timeout_exceeded".

is maybe looking at the problem from the wrong perspective. Maybe it is simply that only one out of two jobs are ever executed. The one that is executed then exceeds the timeout and then the other job is cancelled due to it. This would not explain the case of obsoletes and raise the question why only one of the jobs is assigned to a worker. However, it is very likely what happened because t_finished of both jobs is the same:

openqa=> select id, state, result, reason, t_created, t_started, t_finished from jobs where id in ( 16512096, 16512097 ) order by id;
    id    |   state   |      result      |                    reason                     |      t_created      |      t_started      |     t_finished      
----------+-----------+------------------+-----------------------------------------------+---------------------+---------------------+---------------------
 16512096 | done      | timeout_exceeded | timeout: test execution exceeded MAX_JOB_TIME | 2025-01-20 17:13:24 | 2025-01-20 18:42:47 | 2025-01-20 20:43:24
 16512097 | cancelled | skipped          |                                               | 2025-01-20 17:13:24 |                     | 2025-01-20 20:43:24
(2 rows)

It can of course be that 16512097 was assigned to a worker but the worker went away for some reason. Then the assignment is taken back. However, normally a re-assignment is supposed to happen.

Looks like 16512097 was assigned to a worker. Not sure what happened then but apparently the assignment wasn't effectively happening and then the job was skipped:

$ xzgrep '16512096' /var/log/openqa_scheduler*
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:48.515970Z] [debug] [pid:1734] Assigned job '16512097' to worker ID '2672'
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:48.517437Z] [debug] [pid:1734] [Job#16512097] Prepare for being processed by worker 2672
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:48.544525Z] [debug] [pid:1734] Sent job(s) '16512097' to worker '2672' (worker33:26)
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:48.822243Z] [debug] [pid:1734] Allocated: { job => 16512097, worker => 2672 }
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:43:29.559985Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:43:40.989996Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready
…
/var/log/openqa_scheduler.3.xz:[2025-01-20T20:18:33.044806Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready
/var/log/openqa_scheduler.3.xz:[2025-01-20T20:19:41.399959Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready
/var/log/openqa_scheduler.3.xz:[2025-01-20T20:22:15.657510Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready
/var/log/openqa_scheduler.3.xz:[2025-01-20T20:22:57.245009Z] [debug] [pid:1734] Skipping job 16512097 because dependent jobs are not ready

The other job was assigned as expected:

$ xzgrep '16512096' /var/log/openqa_scheduler*
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:47.093956Z] [debug] [pid:1734] Need to schedule 2 parallel jobs for job 16512096 (with priority 50)
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:47.120683Z] [debug] [pid:1734] Assigned job '16512096' to worker ID '2679' (worker33:33)
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:47.123886Z] [debug] [pid:1734] [Job#16512096] Prepare for being processed by worker 2679
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:47.154420Z] [debug] [pid:1734] Created network for 16512096: 3
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:47.184515Z] [debug] [pid:1734] Sent job(s) '16512096' to worker '2679'
/var/log/openqa_scheduler.3.xz:[2025-01-20T18:42:48.821476Z] [debug] [pid:1734] Allocated: { job => 16512096, worker => 2679 }

I don't think the dependent jobs is not ready due to Minion jobs because scheduling this product shouldn't cause any Minion jobs. We have git_auto_update = no on OSD and the jobs don't specify any Git URLs. I also confirmed that no Minion jobs are created by running openqa_url=http://localhost:9526 distri=sle version=15-SP5 flavor=Server-DVD-Updates test_name=ovs-client ./openqa-schedule-mm-ping-test locally.

I cannot reproduce the problem locally with two workers using PARALLEL_ONE_HOST_ONLY=1. Even if one of the workers doesn't react to the assignment and the half-scheduled cluster needs to be repaired everything works as expected. I can reproduce it if I comment out $self->_pick_siblings_of_running($allocated_jobs, $allocated_workers);. So there is most likely an issue within that function.

EDIT: I can reproduce the following situation that might be exactly what happened in production: When only one of two worker slots accept their assignment and then only another worker slot with mismatching host is available to repair the half-assigned cluster the "Skipping job … because dependent jobs are not ready" message is logged. This is quite confusing. It just means there's no free slot on the required worker host to schedule the job.

PR https://github.com/os-autoinst/openQA/pull/6132 will fix scheduling-related problems that could have lead to this.

This still leaves the obsoletion case but I guess it isn't worth looking into it further.

The documentation about _OBSOLETE=1 says that it obsoletes "jobs in older builds with same DISTRI and VERSION". Judging by the code also FLAVOR and ARCH must match. In our case DISTRI and VERSION are just "opensuse" and "Tumbleweed" anf FLAVOR and ARCH are just "DVD" and "x86_64". That's all very generic. So the jobs are probably obsoleted by a completely unrelated scheduled product. We specify _GROUP=0 but judging by the documentation and code the group is not one of the filter parameters.

It should be easier to find out what scheduled product caused the obsolescence so I created https://github.com/os-autoinst/openQA/pull/6134.

Of course we still need to think how to prevent the obsolescence from happening here.

both https://github.com/os-autoinst/scripts/pull/366 and https://github.com/os-autoinst/openQA/pull/6134 are merged. What's next?