Project

General

Profile

Actions

action #153766

open

[core][sporadic] Handle wild agetty better in tests/network/setup_multimachine.pm

Added by dheidler 3 months ago. Updated about 1 month ago.

Status:
New
Priority:
High
Assignee:
-
Category:
Bugs in existing tests
Target version:
-
Start date:
2024-01-17
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

https://openqa.opensuse.org/tests/3874464
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/159#issuecomment-1894506514

This happens around 2 in 100 multi machine jobs so or 1 in 100 individual jobs.

A wild agetty appeared:

[2024-01-16T11:45:41.007648-05:00] [debug] [pid:10982] <<< testapi::type_string(string="(echo qQf4r; bash -eox pipefail /tmp/scriptqQf4r.sh ; echo SCRIPT_FINISHEDqQf4r-\$?-) | tee /dev/ttyS0\n", max_interval=250,
 wait_screen_change=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2024-01-16T11:45:45.176979-05:00] [debug] [pid:10982] tests/network/setup_multimachine.pm:45 called mm_network::setup_static_mm_network -> lib/mm_network.pm:228 called mm_network::configure_static_dns -> lib/mm_
network.pm:130 called testapi::script_output
[2024-01-16T11:45:45.178288-05:00] [debug] [pid:10982] <<< testapi::wait_serial(regexp="SCRIPT_FINISHEDqQf4r-\\d+-", buffer_size=undef, quiet=undef, expect_not_found=0, record_output=1, no_regex=0, timeout=90)
[2024-01-16T11:45:47.235577-05:00] [debug] [pid:10982] >>> testapi::wait_serial: SCRIPT_FINISHEDqQf4r-\d+-: ok
[2024-01-16T11:45:47.235785-05:00] [debug] [pid:10982] tests/network/setup_multimachine.pm:45 called mm_network::setup_static_mm_network -> lib/mm_network.pm:228 called mm_network::configure_static_dns -> lib/mm_
network.pm:132 called testapi::assert_script_run
[2024-01-16T11:45:47.235910-05:00] [debug] [pid:10982] <<< testapi::assert_script_run(cmd="nmcli connection modify 'Welcome to openSUSE Tumbleweed 20240115 - Kernel 6.6.11-1-default (ttyS0).\n\nens4: 10.0.2.102 f
e80::5054:ff:fe12:2\n\n\nsusetest login: ens4' ipv4.dns '10.0.2.3'", quiet=undef, fail_message="", timeout=90)
[2024-01-16T11:45:47.236029-05:00] [debug] [pid:10982] tests/network/setup_multimachine.pm:45 called mm_network::setup_static_mm_network -> lib/mm_network.pm:228 called mm_network::configure_static_dns -> lib/mm_
network.pm:132 called testapi::assert_script_run
[2024-01-16T11:45:47.236129-05:00] [debug] [pid:10982] <<< testapi::type_string(string="nmcli connection modify 'Welcome to openSUSE Tumbleweed 20240115 - Kernel 6.6.11-1-default (ttyS0).\n\nens4: 10.0.2.102 fe80
::5054:ff:fe12:2\n\n\nsusetest login: ens4' ipv4.dns '10.0.2.3'", max_interval=250, wait_screen_change=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2024-01-16T11:45:53.755719-05:00] [debug] [pid:10982] tests/network/setup_multimachine.pm:45 called mm_network::setup_static_mm_network -> lib/mm_network.pm:228 called mm_network::configure_static_dns -> lib/mm_
network.pm:132 called testapi::assert_script_run
[2024-01-16T11:45:53.758424-05:00] [debug] [pid:10982] <<< testapi::type_string(string="; echo Q94Z9-\$?- > /dev/ttyS0\n", max_interval=250, wait_screen_change=0, wait_still_screen=0, timeout=30, similarity_level
=47)
[2024-01-16T11:45:54.885176-05:00] [debug] [pid:10982] tests/network/setup_multimachine.pm:45 called mm_network::setup_static_mm_network -> lib/mm_network.pm:228 called mm_network::configure_static_dns -> lib/mm_
network.pm:132 called testapi::assert_script_run
[2024-01-16T11:45:54.885778-05:00] [debug] [pid:10982] <<< testapi::wait_serial(regexp=qr/Q94Z9-\d+-/u, quiet=undef, buffer_size=undef, expect_not_found=0, timeout=90, no_regex=0, record_output=undef)
[2024-01-16T11:45:56.941627-05:00] [debug] [pid:10982] >>> testapi::wait_serial: (?^u:Q94Z9-\d+-): ok
[2024-01-16T11:45:56.998284-05:00] [info] [pid:10982] ::: basetest::runtest: # Test died: command 'nmcli connection modify 'Welcome to openSUSE Tumbleweed 20240115 - Kernel 6.6.11-1-default (ttyS0).

  ens4: 10.0.2.102 fe80::5054:ff:fe12:2


  susetest login: ens4' ipv4.dns '10.0.2.3'' failed at /usr/lib/os-autoinst/testapi.pm line 926.
        testapi::assert_script_run("nmcli connection modify 'Welcome to openSUSE Tumbleweed 20240"...) called at opensuse/lib/mm_network.pm line 132
        mm_network::configure_static_dns(HASH(0x563622f61c98), "is_nm", 1) called at opensuse/lib/mm_network.pm line 228
        mm_network::setup_static_mm_network("10.0.2.102/24") called at opensuse/tests/network/setup_multimachine.pm line 45
        setup_multimachine::run(setup_multimachine=HASH(0x563626081bd0)) called at /usr/lib/os-autoinst/basetest.pm line 352
        eval {...} called at /usr/lib/os-autoinst/basetest.pm line 346
        basetest::runtest(setup_multimachine=HASH(0x563626081bd0)) called at /usr/lib/os-autoinst/autotest.pm line 415
        eval {...} called at /usr/lib/os-autoinst/autotest.pm line 415
        autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 272
        eval {...} called at /usr/lib/os-autoinst/autotest.pm line 272
        autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 323
        autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x563627209df0)) called at /usr/lib/perl5/vendor_perl/5.38.2/Mojo/IOLoop/ReadWriteProcess.pm line 329
        eval {...} called at /usr/lib/perl5/vendor_perl/5.38.2/Mojo/IOLoop/ReadWriteProcess.pm line 329
        Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x563627209df0), CODE(0x56362720a060)) called at /usr/lib/perl5/vendor_perl/5.38.2/Mojo/IOLoop/ReadWriteProcess.pm line 492
        Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x563627209df0)) called at /usr/lib/os-autoinst/autotest.pm line 325
        autotest::start_process() called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 94
        OpenQA::Isotovideo::Runner::start_autotest(OpenQA::Isotovideo::Runner=HASH(0x563621c6b1d0)) called at /usr/bin/isotovideo line 192
        eval {...} called at /usr/bin/isotovideo line 181

Related issues 2 (0 open2 closed)

Related to openQA Project - action #138302: Ensure automated openQA tests verify that os-autoinst-setup-multi-machine sets up valid networking size:MResolveddheidler2023-07-192024-01-19

Actions
Related to openQA Project - action #155170: [openqa-in-openqa] [sporadic] test fails in test_running: parallel_failed size:MResolvedybonatakis2024-02-082024-02-29

Actions
Actions #1

Updated by dheidler 3 months ago

  • Related to action #138302: Ensure automated openQA tests verify that os-autoinst-setup-multi-machine sets up valid networking size:M added
Actions #2

Updated by dheidler 3 months ago

  • Description updated (diff)
Actions #3

Updated by dheidler 3 months ago

  • Description updated (diff)
Actions #4

Updated by okurz 3 months ago

  • Project changed from openQA Project to openQA Tests
  • Subject changed from Handle wild agetty better in tests/network/setup_multimachine.pm to [core][sporadic] Handle wild agetty better in tests/network/setup_multimachine.pm
  • Category changed from Regressions/Crashes to Bugs in existing tests
  • Priority changed from Normal to High

@qe-core Fixing this ticket would help us in the tools team to have stable openQA-in-openQA tests

Actions #5

Updated by dheidler 3 months ago

Suggestion: Try to use schedule a lot of https://openqa.opensuse.org/tests/3873052 to reproduce it.

Actions #6

Updated by tinita 2 months ago

  • Related to action #155170: [openqa-in-openqa] [sporadic] test fails in test_running: parallel_failed size:M added
Actions #7

Updated by tinita about 1 month ago

This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

Actions

Also available in: Atom PDF