Actions
action #103581
closedMany jobs on openqa.opensuse.org incomplete in ' timeout: setup exceeded MAX_SETUP_TIME'
Description
Observation¶
There are many jobs on openqa.opensuse.org incomplete in 'timeout: setup exceeded MAX_SETUP_TIME'
The log show:
[2021-12-07T04:47:55.572079+01:00] [info] +++ setup notes +++
[2021-12-07T04:47:55.572463+01:00] [info] Running on openqaworker4:2 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64)
[2021-12-07T04:47:55.577783+01:00] [debug] Found ASSET_1, caching Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box
[2021-12-07T04:47:55.580926+01:00] [info] Downloading Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box, request #495088 sent to Cache Service
[2021-12-07T05:47:55.606967+01:00] [info] +++ worker notes +++
[2021-12-07T05:47:55.607179+01:00] [info] End time: 2021-12-07 04:47:55
[2021-12-07T05:47:55.607254+01:00] [info] Result: timeout
[2021-12-07T05:47:55.619503+01:00] [info] Uploading autoinst-log.txt
Here is an example: https://openqa.opensuse.org/tests/2073891#
Suggestion¶
- If the problem is present on multiple machines rollback the deployment on o3.
- Investigate if recent packages changes, e.g. in os-autoinst or https://build.opensuse.org/package/show/devel:openQA:Leap:15.3/perl-Mojo-IOLoop-ReadWriteProcess can have caused problems
- Investigate the logs on openqaworker4 further for why there were reports about I/O problems
- Ensure consistent stability on our o3 workers
Rollback steps¶
- Enable default worker classes, e.g. "qemu_x86_64" and "qemu_i586" again on openqaworker4 in /etc/openqa/workers.ini
Actions