Project

General

Profile

action #123175

Updated by mkittler about 1 year ago

I have restarted our testing efforts for [kiwi](https://github.com/OSInside/kiwi-functional-tests) which involve downloading a lot of images from OBS and booting them on o3. Unfortunately, o3 has been highly unreliable and often will "fetch" empty disk images or isos which then result in test failures. 

 Examples of such test failures from the kiwi builds: failures: 
 - https://openqa.opensuse.org/tests/3035738 
 - https://openqa.opensuse.org/tests/3035730 
 - https://openqa.opensuse.org/tests/3035728 
 - https://openqa.opensuse.org/tests/3035744 
 - https://openqa.opensuse.org/tests/3035746 
 - https://openqa.opensuse.org/tests/3035750 
 - https://openqa.opensuse.org/tests/3035752 
 - https://openqa.opensuse.org/tests/3035792 
 - https://openqa.opensuse.org/tests/3035790 
 - https://openqa.opensuse.org/tests/3035802 
 - https://openqa.opensuse.org/tests/3035800 
 - https://openqa.opensuse.org/tests/3035798 
 - https://openqa.opensuse.org/tests/3035794 
 - https://openqa.opensuse.org/tests/3035852 
 - https://openqa.opensuse.org/tests/3035842 
 - https://openqa.opensuse.org/tests/3035832 
 - https://openqa.opensuse.org/tests/3035820 
 - https://openqa.opensuse.org/tests/3035862 
 - https://openqa.opensuse.org/tests/3035896 
 - https://openqa.opensuse.org/tests/3035892 
 - https://openqa.opensuse.org/tests/3035868 
 - https://openqa.opensuse.org/tests/3035878 

 These are just the failure from one full test run, which makes o3 unfortunately at the moment very unreliable for testing. Also, due to the nature of the issue, restarting jobs is no good, as o3 will simply restart the test with the same broken image. 

 ## Suggestions 
 * Better log messages by the downloader (to ensure if it really downloaded a zero length file with a 200 status) 
 * Ask when those builds are normally scheduled to be able to investigate the behavior without delay 
 * Run `xzgrep -i -P 'Downloading.*kiwi' /var/log/openqa_gru*` on o3 to find relevant log messages 
 * Check https://openqa.opensuse.org/group_overview/85 for new builds of relevant jobs

Back