action #107941
closedcoordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
coordination #98952: [epic] t/full-stack.t sporadically fails "clickElement: element not interactable" and other errors
[sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M
Description
Observation¶
This problem showed up in https://app.circleci.com/pipelines/github/os-autoinst/openQA/9123/workflows/42eb45bf-126b-43be-97f4-b64af16c27ed/jobs/86387/steps (PR: https://github.com/os-autoinst/openQA/pull/4496) and is again a problem of the os-autoinst test itself - or rather unexpected behavior of the SUT.
Checkout the attached log file and especially to see what's going on. Summary:
sudo poweroff
is not typed completely (onlysudo pow
)- After the prompt, the following message is logged:
udevadm settle - timeout of 5 seconds reached, the event queue contains
/sys/devices/pci0000:00/0000:00:07/virtio3/block/vda (1174)
- Then nothing happens so we see the usual log messages in autoinst-log.txt complaining that the system doesn't shutdown.
Further details¶
- mkittler had to conduct approximediately 60 runs of openQA's fullstack test in circleCI to show the problem once. (And one run of openQA's fullstack test runs the problematic openQA test at least two times.)
- QEMU commmand-line:
/usr/bin/qemu-system-i386 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -m 1024 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -smp 1 -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console1,path=virtio_console1,logfile=virtio_console1.log,logappend=on -device virtconsole,chardev=virtio_console1,name=org.openqa.console.virtio_console1 -chardev socket,path=qmp_socket,server=on,wait=off,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0
Suggestions¶
- Only if we actually see it at least a second time we should look into this again
- Try to send the command again e.g. 2-3 times
Files
Updated by okurz almost 3 years ago
- Tracker changed from coordination to action
- Target version set to Ready
Updated by livdywan almost 3 years ago
- Subject changed from Fullstack test can still fail with "udevadm" log message to Fullstack test can still fail with "udevadm" log message size:M
- Description updated (diff)
- Status changed from New to Workable
We should consider this for a mob session, although it's not appearing very frequently for now
Updated by okurz over 2 years ago
- Subject changed from Fullstack test can still fail with "udevadm" log message size:M to [sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M
- Description updated (diff)
- Priority changed from Normal to Low
- Target version changed from Ready to future
This error was only ever shown once so I updated the description to follow up with the work really only if we have seen this at least a second time and can reproduce it (even if it's hard)
Updated by kraih about 2 years ago
- Assignee set to kraih
This has been in the queue for so long, lets see if it is still relevant.
Updated by kraih about 2 years ago
- Status changed from In Progress to Feedback
I just ran t/full-stack.t
90 times on Circle CI, and the problem has not shown up again. All runs were successful, so the test seems very stable now.