QA » openQA Project » openQA Infrastructure

I rebooted the machine to no avail. I disabled all worker instances for now with systemctl disable --now openqa-worker.target and retriggered all incomplete jobs from today on the openqaworker4 instances looked up from https://openqa.opensuse.org/admin/workers

This triggered tests on other machines but from looking at https://openqa.opensuse.org/tests/ it seems all x86_64 workers are bad. So maybe something in the last update from tonight, however aarch64 seems to be just fine

Trying to revert to an earlier snapshot on w1 for debugging

openqaworker1:~ # snapper ls
Type   | #   | Pre # | Date                    | User | Cleanup | Description             | Userdata     
-------+-----+-------+-------------------------+------+---------+-------------------------+--------------
single | 0   |       |                         | root |         | current                 |              
single | 153 |       | 2019-01-14T00:35:58 CET | root | number  | Snapshot Update of #152 | important=yes
single | 154 |       | 2019-01-15T01:11:08 CET | root | number  | Snapshot Update of #153 | important=yes
single | 155 |       | 2019-01-16T00:59:46 CET | root | number  | Snapshot Update of #154 | important=yes
single | 156 |       | 2019-01-17T01:28:52 CET | root | number  | Snapshot Update of #155 | important=yes
single | 157 |       | 2019-01-18T01:50:56 CET | root | number  | Snapshot Update of #156 | important=yes
single | 158 |       | 2019-01-19T01:54:50 CET | root | number  | Snapshot Update of #157 | important=yes
single | 159 |       | 2019-01-20T01:39:26 CET | root | number  | Snapshot Update of #158 | important=yes
single | 160 |       | 2019-01-21T00:30:04 CET | root | number  | Snapshot Update of #159 | important=yes
single | 161 |       | 2019-01-22T01:45:42 CET | root | number  | Snapshot Update of #160 | important=yes
single | 162 |       | 2019-01-23T01:50:49 CET | root | number  | Snapshot Update of #161 | important=yes
single | 163 |       | 2019-01-24T00:15:09 CET | root |         | Snapshot Update of #162 |              
single | 164 |       | 2019-01-25T00:54:24 CET | root |         | Snapshot Update of #163 |              
openqaworker1:~ # transactional-update rollback 164
transactional-update 2.11 started
Options: rollback 164
Separate /var detected.
Separate /etc detected.
Rollback to snapshot 164...
Please reboot to finish rollback!
openqaworker1:~ # reboot
Connection to openqaworker1 closed by remote host.
Connection to openqaworker1 closed.

Ok, did not help. I think this was the wrong snapshot.

single | 163 |       | 2019-01-24T00:15:09 CET | root |         | Snapshot Update of #162 |              
single | 164 |       | 2019-01-25T00:54:24 CET | root |         | Snapshot Update of #163 |              
openqaworker1:~ # systemctl disable --now openqa-worker.target && transactional-update rollback 163 && reboot
transactional-update 2.11 started
Options: rollback 163
Separate /var detected.
Separate /etc detected.
Rollback to snapshot 163...
Please reboot to finish rollback!
Connection to openqaworker1 closed by remote host.
Connection to openqaworker1 closed.

The latest changes of openQA-worker:

# rpm -q --changelog openQA-worker | less
* Thu Jan 24 2019 coolo@suse.com
- Update to version 4.6.1548347463.2f69b0bf:
  * worker: Fix two perl warnings
  * Use nicer syntax for default vars
  * Add (carried over) bugref to job.done AMQP event

I suspect worker: Fix two perl warnings

I disabled the workers on all w1, w4, imagetester with systemctl disable --now openqa-worker.target (ok, on imagetester I manually called systemctl stop openqa-worker@{1..5} because the target does not seem to be there).

Then I retriggered all incomplete jobs on https://openqa.opensuse.org/tests?&resultfilter=Incomplete , this time manually but I think I already applied a script once. And now openqaworker1 picked them up and it seems to work fine, e.g. https://openqa.opensuse.org/tests/837782 .
In the service journal I can find again the warnings that the above commit tried to fix:

Jan 25 09:36:48 openqaworker1 worker[3143]: Use of uninitialized value in concatenation (.) or string at /usr/share/openqa/script/../lib/OpenQA/Worker/Engines/isotovideo.pm line 245.
Jan 25 09:36:48 openqaworker1 worker[3143]: [info] rsync:
Jan 25 09:36:48 openqaworker1 worker[3143]: Use of uninitialized value $exit in numeric eq (==) at /usr/share/openqa/script/../lib/OpenQA/Worker/Engines/isotovideo.pm line 246.

whereas on w4 that is still affected the journal says

Jan 25 08:59:57 openqaworker4 worker[2427]: Use of uninitialized value $lines[0] in join or string at /usr/share/openqa/script/../lib/OpenQA/Utils.pm line 387.
Jan 25 09:00:02 openqaworker4 worker[2427]: Use of uninitialized value $exit in concatenation (.) or string at /usr/share/openqa/script/../lib/OpenQA/Worker/Engines/isotovideo.pm line 246.

I rolled back w4 and imagester as well. They are happily executing test jobs properly now.

Regarding "How did we come to this?"¶

https://github.com/os-autoinst/openQA/pull/1965 was merged despite tests failing. One reason is certainly that many tests failed because of a new, unspecified dependency "lsof" for a depending package, e.g. see https://travis-ci.org/os-autoinst/openQA/jobs/482477148 . This probably lead mkittler to the conclusion that tests only failed because of that. However https://travis-ci.org/os-autoinst/openQA/jobs/482477148#L1136 was terminated because of a "stalled build" so could point to some other problems here.

Fix¶

https://github.com/os-autoinst/openQA/pull/1972

It's merged, the packages were built for openSUSE Leap 42.3 already and I deployed them on power8. Jobs started fine so I guess we are settled for tomorrow's automatic update.