QA » openQA Project » openQA Infrastructure

I can't establish an SSH connection to OSD. The web UI on the other hand responded promptly.

I've also noticed earlier today problems with the SSH connection to OSD. It took very long to connect and even then it was very slow. At some point it was better again but now I cannot even establish the connection anymore. Maybe these performance problems are actually related to why we have many incomplete jobs. Unfortunately I can't investigate this any further without access to the database. Ok, it works now after waiting 3 minutes or so - just before I wanted to submit this comment.

Almost all of the incompletes were download failures like https://openqa.suse.de/tests/10451514 and https://openqa.suse.de/tests/10451519 ¹. When I remember correctly, a 404 error shouldn't actually lead to an incomplete here. I'm confusing this with https://github.com/os-autoinst/openQA/pull/4844 (which was about Minion jobs for the downloads on the web UI). So supposedly it works as intended (if the assets were really just wrongly specified, all assets I tried really don't exist).

There were also some incompletes because the cache queue was full: https://openqa.suse.de/tests/10449763

For now I have silenced the alert (see https://monitor.qa.suse.de/alerting/silence/06775c82-cefa-4db0-a3ac-f4734410a31d/edit?alertmanager=grafana).

¹ see

`select count(id), array_agg(id), reason from jobs where t_finished >= '2023-02-07T12:00:00' and t_finished <= '2023-02-09T12:00:00' and result = 'incomplete' and clone_id is null group by reason order by count(id) desc;`

We have different issues (the most frequently occurring issue is listed first, nested points are ideas for improvement):

1. openqa-investigate doesn't take private assets into account: The original jobs (before any retry by openQA itself) already lack their chained parent dependency (e.g. https://openqa.suse.de/tests/10451500). They have been created by openqa-investigate. This script apparently blindly re-triggers jobs without checking whether they require a private asset generated by a chained parent job (the job this is cloned by openqa-investigate has its chained parent as expected). So likely something we need to improve in openqa-investigate. It should either:
   1. Determine whether a parent job has generated a private asset and skip the investigation completely (we don't support this case).
   2. Determine whether a parent job has generated a private asset and clone the parent as well (which leads to many duplicate and unnecessarily cloned jobs).
   3. Determine whether a parent job has generated a private asset and adjust all asset-related job settings when cloning the investigation job so the private assets can be found. Either:
      1. Prefix the asset name explicitly in all relevant job settings specifying assets. (Maybe this also needs a slight adjustment on the openQA-side.)
      2. Specify the parent job ID via _START_AFTER when cloning the job. Then the cloned job will reference the parent job of the original one (without re-triggering the parent). This way openQA will be able to make the private asset available.
         1. The side-effect is that the investigation jobs will show up in the original dependency tree (creating one big dependency tree).
         2. This approach will not work when the investigation jobs are triggered on a different instance than the original jobs.
2. openqa-investigate triggers jobs when the asset is long gone: For instance, https://openqa.suse.de/tests/10449641 was triggered for yesterday for the 3 month old (and already archived) job https://openqa.suse.de/tests/9882097. If it would have re-triggered the parent job to re-create the asset it might have even worked. However, it is questionable why this job was triggered in the first place after such a long delay. Almost all jobs I could find via the mentioned SQL query are cases like this (besides 1.).
   1. Skip investigating jobs that are too old.
   2. Skip investigating if not all assets are present anymore.
   3. Clone the parent job as necessary (similarly to 1.2).
3. Other errors, e.g. incompletes due to syntax errors or because of the backend died. Those aren't that many compared to 1 and 2.

Talked about it.

1. openqa-investigate uses openqa-clone-job. openqa-clone-job checks if assets are there when trying to download. But when we call openqa-clone-job … --within-instance there is no download attempt hence the invocation never fails. openqa-clone-job already knows the option --ignore-missing-assets. We should add an additional mode --check-assets or similar to call a HEAD request to check for the presence of assets regardless of --skip-download or not.

2. If the potentially recursive asset check finds out that even the parent(s) does not have any necessary root assets anymore, e.g. the original Tumbleweed DVD ISO file, then report that explicitly to the user in the output of openqa-investigate that ends up in openQA comments

3. If the asset check finds out that an asset is missing but a parent should be able to create it then openqa-clone-job on request or automatically trying to be "smart" should include the asset generation parent in the clone call

4. openqa-clone-job should be smart enough to handle private assets for the option "--within-instance" as we should trust that the private asset actually exists there.

Please put those four ideas into new feature request tickets in "future" or just add to #65271 at least for 2.-4. as they are a single sentence for now.

EDIT: Ok, I just put 2.-4. into #65271