QA » openQA Project

Also see https://github.com/os-autoinst/scripts/pull/20 about that for tracking

The reason "quit" is admittedly not very self-explanatory. However, it has actually only one meaning (the worker received SIGTERM). Although there are no other "quit" reasons we could make it e.g. "quit: associated worker openqaworker-arm-2:17 received SIGTERM". That would be consistent with e.g. "abandoned: associated worker openqaworker-arm-2:17 re-connected but abandoned the job".

Died is a little bit harder. In these cases most likely a problem within isotovideo (e.g. the backend) caused the job to incomplete. Results (up to that point) and logs are usually available. These would provide further information but are usually completely cluttered. So it would be indeed nice to to have the most relevant line from the log added as reason, e.g. died: Error connecting to VNC server <10.161.145.13:5901>: IO::Socket::INET: connect: Connection timed out. Not sure how to achieve that in a generic way, though. Sometimes we might not be able to pin down the problem to a single line anyways or it would just get too long. I suppose it would help if isotovideo (or the subprocess in question) would store its "final fatal error message" at a dedicated place. Of course this fatal error message could as well just be caused by a symptom and the actual problem happened before.

mkittler wrote:

The reason "quit" is admittedly not very self-explanatory. However, it has actually only one meaning (the worker received SIGTERM). Although there are no other "quit" reasons we could make it e.g. "quit: associated worker openqaworker-arm-2:17 received SIGTERM". That would be consistent with e.g. "abandoned: associated worker openqaworker-arm-2:17 re-connected but abandoned the job".

Hm, not sure if "associated worker … received SIGTERM" is really that helpful from user perspective. As I user reading this message I would either not understand it or ask "ok, so what?". Rough idea, how about: "quit: associated worker openqaworker-arm-2:17 shut down, job was cloned" ?

Died is a little bit harder. In these cases most likely a problem within isotovideo (e.g. the backend) caused the job to incomplete. Results (up to that point) and logs are usually available. […] Sometimes we might not be able to pin down the problem to a single line anyways or it would just get too long.

Yes, how about just: "died: See log output for details". For this I created a simple https://github.com/os-autoinst/openQA/pull/2854

For "quit" I did not easily understand where I could add more information to the string.

Rough idea, how about: "quit: associated worker openqaworker-arm-2:17 shut down, job was cloned" ?

I don't like the idea.

1. "job was cloned" is redundant (there's the clone_id in the DB and in the web UI) and maybe not even correct (at the point the reason is recorded we might not whether we will successfully clone that job).
2. "shut down" is more vague. It doesn't tell us the reason for the shut down, e.g. one might ask whether the worker crashed or why it decided to shut down. If "SIGTERM" is too technical what about "[...] has been stopped or restarted"? The passive makes at least clear that the worker merely reacted but did not initiate it.

Yes, how about just: "died: See log output for details".

I suppose people are used to look into the logs anyways but I like it for consistency.

For "quit" I did not easily understand where I could add more information to the string.

Here's a draft PR: https://github.com/os-autoinst/openQA/pull/2855

https://github.com/os-autoinst/openQA/pull/2854 merged, https://github.com/os-autoinst/openQA/pull/2855 failed on scalability test failure. Either needs another fix or a rebase.

That's the list of jobs incompleted with just died we've still got on OSD since the 1st August:

select jobs.id,t_started,t_finished,test,reason,result,concat('https://openqa.suse.de/tests/', jobs.id) as url from jobs where result = 'incomplete' and reason = 'died' and t_finished >= '2020-08-01' order by t_finished desc;

Here the most interesting cases:

1. # Test died: script timeout: zypper lifecycle at /usr/lib/os-autoinst/testapi.pm line 1104.: We should likely treat a timeout within script_output as a test failure. So the job should not be incomplete at all.
2. Could not find 'validate_multipath' in complete history at /usr/lib/os-autoinst/OpenQA/Isotovideo/Utils.pm line 85.: Errors from the Git helpers should likely also be serialized with bmwqemu::serialize_state.
3. needles_dir not found: /var/lib/openqa/pool/24/os-autoinst-distri-opensuse/os-autoinst-distri-opensuse/products/sle/needles (check vars.json?) at /usr/lib/os-autoinst/needle.pm line 330.: same as 2.

Other failures look identical or there's no autoinst-log.txt anymore so it is hard to tell what the problem was.

Maybe we should also add a general die-handler to isotovideo (and not just the backend).

PR: https://github.com/os-autoinst/openQA/pull/3359

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: kde
https://openqa.opensuse.org/tests/1484527

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed