action #12154
closedaarch64 worker I/O problems
0%
Description
observation¶
https://openqa.suse.de/tests/400707/file/autoinst-log.txt --> "Could not write to image qemuscreenshot/shot-0000001830.png.tmp"
also see related tickets:
steps to reproduce¶
anything on aarch64, probably when many jobs are running in parallel
problem¶
H1. aarch64 worker I/O problems
suggestion¶
try to reproduce
workaround¶
retrigger jobs
Updated by okurz about 8 years ago
- Copied from action #12014: QEMU: usb-kbd: warning: key event queue full added
Updated by okurz about 8 years ago
- Status changed from New to In Progress
Dirk Müller and Richard Brown did some changes to our aarch64 worker. https://openqa.suse.de/tests/403002 is a passed job cloned from one that previously failed because of worker problems. I did not find other good candidates for aarch64 to retrigger and see about the result so we need to wait for more builds to trigger on aarch64, too, and observe if they show up similar symptoms.
Updated by okurz about 8 years ago
- Assignee changed from RBrownSUSE to okurz
Retriggered the aforementioned job for 20 times as 403141, 403142, 403143, 403144, 403145, 403146, 403147, 403148, 403149, 403150, 403151, 403152, 403153, 403155, 403157, 403158, 403159, 403160, 403161, 403162, 403163, 403164, 403165, 403166, 403167, 403168, 403169, 403170, 403171, 403172, 403173, 403174, 403175, 403176, 403177, 403178, 403179, 403180, 403181, 403182
Let's see the results as soon as they finished. if they all work, we can close this and the subticket, else need to distinguish between "keyboard queue event full", "worker out of memory", "other I/O problems", etc.
To trigger twenty times I did
for i in {1..20} ; do /local/openQA/script/clone_job.pl --host https://openqa.suse.de --apikey XXX --apisecret XXX --from https://openqa.suse.de 403002 ; done
Checking for result should work with:
for i in 403141, 403142, 403143, 403144, 403145, 403146, 403147, 403148, 403149, 403150, 403151, 403152, 403153, 403155, 403157, 403158, 403159, 403160, 403161, 403162, 403163, 403164, 403165, 403166, 403167, 403168, 403169, 403170, 403171, 403172, 403173, 403174, 403175, 403176, 403177, 403178, 403179, 403180, 403181, 403182; do openqa_client_osd jobs/${i/,/} | grep result ; done | uniq -c
as I use it with my "client wrapper script" openqa_client_osd (as "client" but with api credentials and from specified).
Updated by okurz about 8 years ago
- Assignee deleted (
okurz)
result from previous experiment: 8 runs passed, 3 failed, some where obsoleted by more recent builds
results from "failed":
- https://openqa.suse.de/tests/403147/modules/first_boot/steps/5 times out booting into the installed system which might be acceptable aarch64 performance, no further action done so far
- https://openqa.suse.de/tests/403150/modules/install_and_reboot/steps/21 and https://openqa.suse.de/tests/403153 fail while installing different packages, bug filed -> https://bugzilla.suse.com/show_bug.cgi?id=982242
conclusion: The latter might be related to worker performance problems
unassigning, collecting feedback
Updated by mgriessmeier almost 8 years ago
Build1999 [BETA5]:
https://openqa.suse.de/tests/481511
-> keyboard queue full -> retriggered
Updated by mkravec almost 8 years ago
Build2155:
QEMU: usb-kbd: warning: key event queue full -> retriggered
https://openqa.suse.de/tests/599160#step/no_perl_bootloader/1
Updated by coolo over 7 years ago
- Status changed from In Progress to Closed
The remaining issues should better be handled as part of 14972