Project

General

Profile

Actions

action #107941

closed

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #98952: [epic] t/full-stack.t sporadically fails "clickElement: element not interactable" and other errors

[sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M

Added by mkittler over 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2022-03-07
Due date:
% Done:

0%

Estimated time:

Description

Observation

This problem showed up in https://app.circleci.com/pipelines/github/os-autoinst/openQA/9123/workflows/42eb45bf-126b-43be-97f4-b64af16c27ed/jobs/86387/steps (PR: https://github.com/os-autoinst/openQA/pull/4496) and is again a problem of the os-autoinst test itself - or rather unexpected behavior of the SUT.

Checkout the attached log file and especially to see what's going on. Summary:

  • sudo poweroff is not typed completely (only sudo pow)
  • After the prompt, the following message is logged:
udevadm settle - timeout of 5 seconds reached, the event queue contains
  /sys/devices/pci0000:00/0000:00:07/virtio3/block/vda (1174)
  • Then nothing happens so we see the usual log messages in autoinst-log.txt complaining that the system doesn't shutdown.

Further details

  • mkittler had to conduct approximediately 60 runs of openQA's fullstack test in circleCI to show the problem once. (And one run of openQA's fullstack test runs the problematic openQA test at least two times.)
  • QEMU commmand-line: /usr/bin/qemu-system-i386 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -m 1024 -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -smp 1 -no-shutdown -vnc :91,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console1,path=virtio_console1,logfile=virtio_console1.log,logappend=on -device virtconsole,chardev=virtio_console1,name=org.openqa.console.virtio_console1 -chardev socket,path=qmp_socket,server=on,wait=off,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/hd0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0,file=hd0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0,serial=hd0 -blockdev driver=file,node-name=cd0-overlay0-file,filename=/home/squamata/project/test-results/fullstack/full-stack.d/openqa/pool/1/raid/cd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=cd0-overlay0,file=cd0-overlay0-file,cache.no-flush=on -device scsi-cd,id=cd0-device,drive=cd0-overlay0,serial=cd0

Suggestions

  • Only if we actually see it at least a second time we should look into this again
  • Try to send the command again e.g. 2-3 times

Files

autoinst-log.txt (22.7 KB) autoinst-log.txt mkittler, 2022-03-07 14:46
video.ogg (171 KB) video.ogg mkittler, 2022-03-07 14:46
Actions #1

Updated by mkittler over 2 years ago

  • Description updated (diff)
Actions #2

Updated by mkittler over 2 years ago

  • Description updated (diff)
Actions #3

Updated by okurz over 2 years ago

  • Tracker changed from coordination to action
  • Target version set to Ready
Actions #4

Updated by livdywan over 2 years ago

  • Subject changed from Fullstack test can still fail with "udevadm" log message to Fullstack test can still fail with "udevadm" log message size:M
  • Description updated (diff)
  • Status changed from New to Workable

We should consider this for a mob session, although it's not appearing very frequently for now

Actions #5

Updated by okurz over 2 years ago

  • Subject changed from Fullstack test can still fail with "udevadm" log message size:M to [sporadic] openQA Fullstack test t/full-stack.t can still fail with "udevadm" log message size:M
  • Description updated (diff)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

This error was only ever shown once so I updated the description to follow up with the work really only if we have seen this at least a second time and can reproduce it (even if it's hard)

Actions #6

Updated by okurz about 2 years ago

  • Target version changed from future to Ready
Actions #7

Updated by kraih over 1 year ago

  • Assignee set to kraih

This has been in the queue for so long, lets see if it is still relevant.

Actions #8

Updated by kraih over 1 year ago

  • Status changed from Workable to In Progress
Actions #9

Updated by kraih over 1 year ago

  • Status changed from In Progress to Feedback

I just ran t/full-stack.t 90 times on Circle CI, and the problem has not shown up again. All runs were successful, so the test seems very stable now.

Actions #10

Updated by kraih over 1 year ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF