QA » openQA Project

When looking into failed jobs with parallel dependencies since Saturday it doesn't look too bad in absolute figures:

openqa=> select distinct count(jobs.id), array_agg(jobs.id), (select name from job_groups where id = group_id), (array_agg(test))[1] as example_test from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) and clone_id is null where dependency = 2 and t_finished >= '2024-01-20' and result in ('failed') and test not like '%:investigate:%' group by group_id order by count(jobs.id) desc;
 count |                                                                                  array_agg                                                                                   |                  name                  |                   example_test                    
-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------+---------------------------------------------------
    19 | {13314890,13302354,13314880,13314892,13314888,13302360,13302352,13302350,13302310,13314874,13314878,13314882,13302337,13302339,13302362,13302356,13314870,13302341,13314872} | YaST Maintenance Updates - Development | mru-iscsi_client_normal_auth_backstore_fileio_dev
     5 | {13310295,13310164,13310164,13310295,13308014}                                                                                                                               | Maintenance: SLE 15 SP5 HA Incidents   | qam_ha_rolling_upgrade_migration_supportserver
     3 | {13302400,13306361,13302398}                                                                                                                                                 | YaST Maintenance Updates               | mru-iscsi_server_normal_auth_backstore_hdd
     3 | {13311675,13311784,13311713}                                                                                                                                                 | Maintenance - QR - SLE15SP5-SAP        | ha_qdevice_node2
     2 | {13294837,13297632}                                                                                                                                                          | Maintenance: SLE 15 SP4 HA Incidents   | qam_ha_rolling_update_node02
     2 | {13299897,13312377}                                                                                                                                                          | JeOS: Development                      | jeos-nfs-client
     2 | {13314918,13309818}                                                                                                                                                          | Test Security                          | cc_ipsec_client
     1 | {13311651}                                                                                                                                                                   | Maintenance - QR - SLE15SP5-Security   | fips_env_stunnel_client
     1 | {13315685}                                                                                                                                                                   | Maintenance: SLE 15 SP2 HA Incidents   | qam_ha_rolling_update_node02
     1 | {13316822}                                                                                                                                                                   | HA  Development                        | ha_hawk_haproxy_node02_test
     1 | {13316842}

When also considering parallel failed the numbers double or triple depending on the number of jobs in the scenario but I guess it is more useful to think in terms of scenarios (and thus only querying for failed).

But some of the failures look in fact like the network within the SUTs was broken, e.g. https://openqa.suse.de/tests/13314878#step/patch_and_reboot/196. Considering we currently still disrupt the GRE network when booting or shutting down a worker it is unfortunately expected that we currently see a small number of jobs like this. (Of course there are also failures which have likely a non-network related cause, e.g. https://openqa.suse.de/tests/13310295#step/setup/54 and https://openqa.suse.de/tests/13294837#step/suseconnect_scc/37.)

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1098 merged, deploying. Waiting for deployment and update of data in monitoring.

EDIT: We talked about the ecountered failed jobs and we agreed that https://openqa.suse.de/tests/13310295#step/setup/54 is a red herring, a test job triggered by emiura

As discussed in daily the problem is because we only consider a timeframe of 24h so it can happen that there are for example just 2 jobs scheduled leading to bad statistics. Instead we could change to number of jobs, e.g. in monitoring/telegraf/telegraf-webui.conf go from

sqlquery="with mm_jobs as (select …) where t_created >= (select timezone('UTC', now()) - interval '24 hour') … as ratio_mm from mm_jobs group by mm_jobs.result"

sqlquery="with mm_jobs as (select …) where … as ratio_mm from mm_jobs group by mm_jobs.result ordery by id limit 1000"

MR for that: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1103