action #103581
Updated by okurz over 2 years ago
## Observation There are many jobs on openqa.opensuse.org incomplete in 'timeout: setup exceeded MAX_SETUP_TIME' The log show: ``` [2021-12-07T04:47:55.572079+01:00] [info] +++ setup notes +++ [2021-12-07T04:47:55.572463+01:00] [info] Running on openqaworker4:2 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64) [2021-12-07T04:47:55.577783+01:00] [debug] Found ASSET_1, caching Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box [2021-12-07T04:47:55.580926+01:00] [info] Downloading Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box, request #495088 sent to Cache Service [2021-12-07T05:47:55.606967+01:00] [info] +++ worker notes +++ [2021-12-07T05:47:55.607179+01:00] [info] End time: 2021-12-07 04:47:55 [2021-12-07T05:47:55.607254+01:00] [info] Result: timeout [2021-12-07T05:47:55.619503+01:00] [info] Uploading autoinst-log.txt ``` Here is an example: https://openqa.opensuse.org/tests/2073891# ## Suggestion * If the problem is present on multiple machines if need to rollback the deployment on o3. * Investigate if recent packages changes, e.g. in os-autoinst or https://build.opensuse.org/package/show/devel:openQA:Leap:15.3/perl-Mojo-IOLoop-ReadWriteProcess can have caused problems * Investigate the logs on openqaworker4 further for why there were reports about I/O problems * Ensure consistent stability on our o3 workers ## Rollback steps * Enable default worker classes, e.g. "qemu_x86_64" and "qemu_i586" again on openqaworker4 in /etc/openqa/workers.ini