action #174583
Updated by okurz about 1 month ago
### Observation
The pipeline is failing because the openQA jobs got obsoleted:
See: https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3562638
```
{"blocked_by_id":null,"id":4713396,"result":"obsoleted","state":"done"}
{"blocked_by_id":null,"id":4713397,"result":"obsoleted","state":"done"}
```
for unknown reason as we don't trigger with OBSOLETE and that should not be default according to openQA documentation
The multimachine case looks a bit more involved e.g. https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3625091 :
```
{"blocked_by_id":null,"id":16374878,"result":"skipped","state":"cancelled"}
1490{"blocked_by_id":null,"id":16374879,"result":"timeout_exceeded","state":"done"}
```
### Acceptance Criteria
* **AC1**: Unfinished Obsoleted jobs don't cause failures in GitLab pipelines
## Suggestions
* ~~Verify if this is a specific worker or workers and take them out of production~~
* ~~Consider restarting affected jobs~~
* ~~An An "obsolete" should be considered part of expected behavior. How about a new openQA API route to follow job obsolescence?~~ -> handled in #175299 obsolescence?
* ~~Ignore Ignore the case of "obsoleted" jobs as the pipeline runs frequently enough anyway.~~ check why jobs ended up as obsolete even though scripts-ci don't trigger with obsoletion anyway.
* Check whether we cancel the full parallel cluster in case a job in it is cancelled/obsoleted as we also saw jobs with parallel dependencies ending up with the result "timeout_exceeded".
* Treat skipped/cancelled the same as obsoleted (and ignore it)
* Ensure this is logged in case it is not always the case
### Mitigations
* ~**DONE** Pause [affected pipelines on GitLab](https://gitlab.suse.de/openqa/scripts-ci/-/pipeline_schedules) i.e. openqa-schedule-mm-ping-test o3/osd~