Project

General

Profile

action #107941

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #98952: [epic] t/full-stack.t sporadically fails "clickElement: element not interactable" and other errors

[sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M

Added by mkittler 4 months ago. Updated about 1 month ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Concrete Bugs
Target version:
Start date:
2022-03-07
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

This problem showed up in https://app.circleci.com/pipelines/github/os-autoinst/openQA/9123/workflows/42eb45bf-126b-43be-97f4-b64af16c27ed/jobs/86387/steps (PR: https://github.com/os-autoinst/openQA/pull/4496) and is again a problem of the os-autoinst test itself - or rather unexpected behavior of the SUT.

Checkout the attached log file and especially to see what's going on. Summary:

  • sudo poweroff is not typed completely (only sudo pow)
  • After the prompt, the following message is logged:
udevadm settle - timeout of 5 seconds reached, the event queue contains
  /sys/devices/pci0000:00/0000:00:07/virtio3/block/vda (1174)
  • Then nothing happens so we see the usual log messages in autoinst-log.txt complaining that the system doesn't shutdown.

Further details

  • mkittler had to conduct approximediately 60 runs of openQA's fullstack test in circleCI to show the problem once. (And one run of openQA's fullstack test runs the problematic openQA test at least two times.)
  • QEMU commmand-line: /usr/bin/qemu-system-i386 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -m 1024 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -smp 1 -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console1,path=virtio_console1,logfile=virtio_console1.log,logappend=on -device virtconsole,chardev=virtio_console1,name=org.openqa.console.virtio_console1 -chardev socket,path=qmp_socket,server=on,wait=off,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0

Suggestions

  • Only if we actually see it at least a second time we should look into this again
  • Try to send the command again e.g. 2-3 times
autoinst-log.txt (22.7 KB) autoinst-log.txt mkittler, 2022-03-07 14:46
video.ogg (171 KB) video.ogg mkittler, 2022-03-07 14:46

History

#1 Updated by mkittler 4 months ago

  • Description updated (diff)

#2 Updated by mkittler 4 months ago

  • Description updated (diff)

#3 Updated by okurz 4 months ago

  • Tracker changed from coordination to action
  • Target version set to Ready

#4 Updated by cdywan 4 months ago

  • Subject changed from Fullstack test can still fail with "udevadm" log message to Fullstack test can still fail with "udevadm" log message size:M
  • Description updated (diff)
  • Status changed from New to Workable

We should consider this for a mob session, although it's not appearing very frequently for now

#5 Updated by okurz 3 months ago

  • Subject changed from Fullstack test can still fail with "udevadm" log message size:M to [sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M
  • Description updated (diff)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

This error was only ever shown once so I updated the description to follow up with the work really only if we have seen this at least a second time and can reproduce it (even if it's hard)

#6 Updated by okurz about 1 month ago

  • Target version changed from future to Ready

Also available in: Atom PDF