QA » openQA Project

Observation¶

Date: Sat, 9 Mar 2024 03:49:48 +0100 (CET)

See <http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/1001/display/redirect>

Changes:


------------------------------------------
[...truncated 4.20 MiB...]
  <result project="devel:openQA:tested" repository="openSUSE_Factory" arch="x86_64" code="blocked" state="blocked">
    <status package="openQA" code="blocked">
+ echo 'Waiting while openQA is in progress'
Waiting while openQA is in progress

...
4.6.1709822711.90519fe6
4.6.1709822711.90519fe6
4.6.1709822711.90519fe6
4.6.1709822711.90519fe6' openSUSE:Factory
Server returned an error: HTTP Error 503: Service Unavailable

Suggestions¶

• Use https://build.opensuse.org/package/show/openSUSE:Factory/retry in the according script from github.com/os-autoinst/scripts/

I assume os-autoinst-obs-auto-submit:38 is the relevant code. Let's see if I can propose a trivial fix.

Setting due date based on mean cycle time of SUSE QE Tools

https://github.com/os-autoinst/scripts/pull/300 correction, it is the osc call

Merged. Let's see if it works fine. Would probably resolve it soon since we likely won't see another outage.

Still need some update though, see the error:

+ retry -e osc co --server-side-source-service-files devel:openQA/openQA
retry: unrecognized option '--server-side-source-service-files'
usage: /usr/bin/retry [options] [cmd...]
options:
    -h,--help                   Show this help
    -r,--retries=RETRIES        How many retries to do on command failure after
                                the initial try. Defaults to 3.
    -s,--sleep=SLEEP            How many seconds to sleep between retries.
                                Defaults to 3 seconds.
    -e,--exponential[=FACTOR]   Enable simple exponential back-off algorithm.
                                Disabled by default, factor defaults to 2
                                (binary exponential back-off). 

New PR was created and merged and verified in jenkins. There is still one problematic entry:

+++ retry -e -- osc cat openSUSE:Factory/openQA/openQA.changes
+++ grep 'Update to version'
+++ head -n1
Retrying up to 3 more times after sleeping 3s …
Retrying up to 2 more times after sleeping 6s …
Retrying up to 1 more times after sleeping 12s …

there shouldn't have been a retry here. Apparently there is a SIGPIPE due to the head. Try to reproduce with

retry -r0 -e -- osc cat openSUSE:Factory/openQA/openQA.changes | grep 'Update to version' | head -n1; echo "${PIPESTATUS[@]}"
- Update to version 4.6.1710762624.7d0dd225:
1 141 0

The grep 'Update to version' | head -n1 can actually be simplified to grep -m1 'Update to version' but that does not yet fix the original problem:

retry -r0 -e -- osc cat openSUSE:Factory/openQA/openQA.changes | grep -m1 'Update to version'; echo "${PIPESTATUS[@]}"
- Update to version 4.6.1710762624.7d0dd225:
1 0

which gets rid of the sigpipe of grep but keeps the failure of osc cat. Then I found one other possibility:

grep -m1 'Update to version' <(retry -r0 -e -- osc cat openSUSE:Factory/openQA/openQA.changes); echo "${PIPESTATUS[@]}"
- Update to version 4.6.1710762624.7d0dd225:
0

created https://github.com/os-autoinst/scripts/pull/306

I wonder why http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/1010/console still mentions multiple "Retrying", looking into that. and the Retrying up to 2 more times after sleeping 6s … line is doubled. there seems to be some retry process going on in the background as there are lines like "Retrying up to 1 more times after sleeping 12s …" just intermixed with other content. ok, the process redirection seems to be a bad idea.

https://github.com/os-autoinst/scripts/pull/310

merged, http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/1011/console looks better now but in the end still failed.

https://github.com/os-autoinst/scripts/pull/312

Setting due date based on mean cycle time of SUSE QE Tools

http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/1012/console succeeded now and no mentions of "Retrying"