Project

General

Profile

action #103581

Updated by okurz almost 3 years ago

## Observation 

 There are many jobs on openqa.opensuse.org incomplete in 'timeout: setup exceeded MAX_SETUP_TIME' 
 The log show: 
 ``` 
 [2021-12-07T04:47:55.572079+01:00] [info] +++ setup notes +++ 
 [2021-12-07T04:47:55.572463+01:00] [info] Running on openqaworker4:2 (Linux 5.3.18-59.34-default #1 SMP Thu Nov 11 12:18:45 UTC 2021 (a2a53aa) x86_64) 
 [2021-12-07T04:47:55.577783+01:00] [debug] Found ASSET_1, caching Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box 
 [2021-12-07T04:47:55.580926+01:00] [info] Downloading Tumbleweed.x86_64-1.0-libvirt-Snapshot20211205.vagrant.libvirt.box, request #495088 sent to Cache Service 
 [2021-12-07T05:47:55.606967+01:00] [info] +++ worker notes +++ 
 [2021-12-07T05:47:55.607179+01:00] [info] End time: 2021-12-07 04:47:55 
 [2021-12-07T05:47:55.607254+01:00] [info] Result: timeout 
 [2021-12-07T05:47:55.619503+01:00] [info] Uploading autoinst-log.txt 
 ``` 

 Here is an example: https://openqa.opensuse.org/tests/2073891# 

 ## Suggestion 
 * If the problem is present on multiple machines if need to rollback the deployment on o3. 
 * Investigate if recent packages changes, e.g. in os-autoinst or https://build.opensuse.org/package/show/devel:openQA:Leap:15.3/perl-Mojo-IOLoop-ReadWriteProcess can have caused problems 
 * Investigate the logs on openqaworker4 further for why there were reports about I/O problems 
 * Ensure consistent stability on our o3 workers 

 ## Rollback steps 
 * Enable default worker classes, e.g. "qemu_x86_64" and "qemu_i586" again on openqaworker4 in /etc/openqa/workers.ini

Back