action #154624
closedopenQA Project - coordination #112862: [saga][epic] Future ideas for easy multi-machine handling: MM-tests as first-class citizens
openQA Project - coordination #111929: [epic] Stable multi-machine tests covering multiple physical workers
Periodically running simple ping-check multi-machine tests on x86_64 covering multiple physical hosts on OSD alerting tools team on failures size:M
0%
Description
Motivation¶
In cases like #154552 multi-machine issues (still) happen and while we monitor multi-machine test results there are cases where users notify us about problems that we don't see in our monitoring. Because we now (#138302) have a good simple ping-check multi-machine test scenario created by dheidler we can use that scenario similar to openQA-in-openQA tests running periodically very often and whenever that scenario fails - because it's so simple likely the cause is multi-machine infrastructure related problems we want to know about - then alert the tools team directly, e.g. email to Slack #team-qa-tools or something, using openqa-label-known-issues
Acceptance criteria¶
- AC1: simple ping-check multi-machine tests executed on x86_64 on OSD periodically covering multiple physical hosts
- AC2: The tools team is alerted directly if those tests fail
Suggestions¶
- Read #138302 where dheidler added the simple ping-check for openQA-in-openQA tests
- The scenario can be in a new job group or just groupless
- Think about how to trigger periodically, possibly gitlab CI pipeline?
- Similar to wicked tests
- Ensure the alerting, possibly use https://github.com/os-autoinst/scripts/?tab=readme-ov-file#unreviewed-issues
Updated by okurz 3 months ago
- Copied from action #154552: [ppc64le] test fails in iscsi_client - zypper reports Error Message: Could not resolve host: openqa.suse.de added
Updated by okurz 3 months ago
- Subject changed from Periodically running simple ping-check multi-machine tests on x86_64+ppc64le covering multiple physical hosts on OSD alerting tools team on failures to Periodically running simple ping-check multi-machine tests on x86_64+ppc64le covering multiple physical hosts on OSD alerting tools team on failures size:M
- Status changed from New to Workable
Updated by okurz 3 months ago
- Related to action #154021: [alert] Ratio of not restarted multi-machine tests by result added
Updated by okurz 3 months ago
- Copied to action #155200: Periodically running simple ping-check multi-machine tests on ppc64le covering multiple physical hosts on OSD alerting tools team on failures size:M added
Updated by okurz 3 months ago
- Subject changed from Periodically running simple ping-check multi-machine tests on x86_64+ppc64le covering multiple physical hosts on OSD alerting tools team on failures size:M to Periodically running simple ping-check multi-machine tests on x86_64 covering multiple physical hosts on OSD alerting tools team on failures size:M
- Description updated (diff)
Updated by jbaier_cz 3 months ago
- Related to action #155278: o3 aarch64 multi-machine tests on openqaworker-arm21 and 22 fail to resolve codecs.opensuse.org size:M added
Updated by jbaier_cz 3 months ago
- Status changed from In Progress to Workable
I created a simple script https://github.com/os-autoinst/scripts/pull/290, which could be executed without any parameters to schedule a multi-machine ping test on o3: https://openqa.opensuse.org/tests/3933553; with some non-default variables, for example openqa_url=openqa.suse.de distri=sle version=15-SP5 flavor=Server-DVD-Updates test_name=ovs-client it could be also used on osd: https://openqa.suse.de/tests/13503761. As this is using openqa-cli schedule --monitor, it should be usable within any other CI/script. My plan is to integrate it into https://gitlab.suse.de/openqa/scripts-ci, where we already have notifications on failed pipeline and where we can easily schedule/pause the execution.
Updated by jbaier_cz 2 months ago
Support for using openqa-cli (creating config file) added in https://gitlab.suse.de/openqa/scripts-ci/-/merge_requests/3
Updated by okurz 2 months ago
https://gitlab.suse.de/openqa/scripts-ci/-/merge_requests/3 merged. What's the next step?
Updated by jbaier_cz 2 months ago
I created new schedules in https://gitlab.suse.de/openqa/scripts-ci/-/pipeline_schedules; and now, when the MR is merged, it should be able to see first result for osd: https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2290689 (and also o3: https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2290690).
If I get a successful pipeline run, I can enable periodic run for osd and we are done.
Updated by jbaier_cz 2 months ago
- Status changed from In Progress to Resolved
Schedule is working (for start, there is a test run every hour); in case of failure, we will be notified via e-mail. The concrete worker assignment, if needed, can be adjusted by editing the YAML schedule within the openqa-schedule-mm-ping-test script.