action #123175
Updated by mkittler almost 2 years ago
I have restarted our testing efforts for [kiwi](https://github.com/OSInside/kiwi-functional-tests) which involve downloading a lot of images from OBS and booting them on o3. Unfortunately, o3 has been highly unreliable and often will "fetch" empty disk images or isos which then result in test failures. Examples of such test failures from the kiwi builds: - https://openqa.opensuse.org/tests/3035738 - https://openqa.opensuse.org/tests/3035730 - https://openqa.opensuse.org/tests/3035728 - https://openqa.opensuse.org/tests/3035744 - https://openqa.opensuse.org/tests/3035746 - https://openqa.opensuse.org/tests/3035750 - https://openqa.opensuse.org/tests/3035752 - https://openqa.opensuse.org/tests/3035792 - https://openqa.opensuse.org/tests/3035790 - https://openqa.opensuse.org/tests/3035802 - https://openqa.opensuse.org/tests/3035800 - https://openqa.opensuse.org/tests/3035798 - https://openqa.opensuse.org/tests/3035794 - https://openqa.opensuse.org/tests/3035852 - https://openqa.opensuse.org/tests/3035842 - https://openqa.opensuse.org/tests/3035832 - https://openqa.opensuse.org/tests/3035820 - https://openqa.opensuse.org/tests/3035862 - https://openqa.opensuse.org/tests/3035896 - https://openqa.opensuse.org/tests/3035892 - https://openqa.opensuse.org/tests/3035868 - https://openqa.opensuse.org/tests/3035878 These are just the failure from one full test run, which makes o3 unfortunately at the moment very unreliable for testing. Also, due to the nature of the issue, restarting jobs is no good, as o3 will simply restart the test with the same broken image. ## Suggestions * Better log messages by the downloader (to ensure if it really downloaded a zero length file with a 200 status) * Ask when those builds are normally scheduled to be able to investigate the behavior without delay * Check https://openqa.opensuse.org/group_overview/85 for new builds of relevant jobs * Check the Minion dashboard for relevant download jobs (https://openqa.opensuse.org/minion/jobs?task=download_asset) after jobs have been scheduled * Run `xzgrep -i -P 'Downloading.*kiwi' /var/log/openqa_gru*` on o3 to find relevant log messages * Try to reproduce download problems using curl or by restarting Check https://openqa.opensuse.org/group_overview/85 for new builds of relevant download jobs in the Minion dashboard