QA » openQA Project » openQA Tests » qe-yam

this looks like an sporadic issue, but I think it's worth investigating since assert_script_run('curl -O ' . data_url('textinfo')); belongs in the backend and that's a critical thing :)

# Test died: command 'curl -O http://10.0.2.2:20063/CldQCbjudrGe1EfF/data/textinfo' failed at /usr/lib/os-autoinst/testapi.pm line 953.
    testapi::_handle_script_run_ret(7, "curl -O http://10.0.2.2:20063/CldQCbjudrGe1EfF/data/textinfo", "fail_message", "", "timeout", 90, "quiet", undef) called at /usr/lib/os-autoinst/testapi.pm line 991
    testapi::assert_script_run("curl -O http://10.0.2.2:20063/CldQCbjudrGe1EfF/data/textinfo") called at microos/tests/console/textinfo.pm line 39
    textinfo::run(textinfo=HASH(0x55c8af23c578)) called at /usr/lib/os-autoinst/basetest.pm line 360
    eval {...} called at /usr/lib/os-autoinst/basetest.pm line 354
    basetest::runtest(textinfo=HASH(0x55c8af23c578)) called at /usr/lib/os-autoinst/autotest.pm line 372
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 372
    autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 242
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 242
    autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 296
    autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x55c8b0bc8790)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x55c8b0bc8790), CODE(0x55c8b0c86640)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 488
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x55c8b0bc8790)) called at /usr/lib/os-autoinst/autotest.pm line 298
    autotest::start_process() called at /usr/bin/isotovideo line 256

In this test the system is setup by a remote target. I don't think this should make a difference. However, it doesn't look like there's any attempt made to wait until the network is ready on the machine. It just waits for the login prompt and than immediately switches to the serial console attempting the download:

     $self->select_serial_terminal;
     assert_script_run('curl -O ' . data_url('textinfo'));

I find it a bit weird that there actually is a test module networking to test whether networking works but it only runs after textinfo and doesn't wait until the network is actually ready (with some timeout). It would make more sense to swap the order of those test modules and wait until the network is ready within networking.

I'm actually pretty sure that in these failures the networking just isn't ready. That's because the serial log shows that only the IPv6 address is assigned in the first place and the IPv4 address (which we rely on here) is only assigned later:

[  406.246840][    T1] reboot: Restarting system
[   15.567090][    T1] systemd[1]: Failed to transition into init label 'system_u:system_r:init_t:s0', ignoring.



Welcome to openSUSE MicroOS (x86_64) - Kernel 5.16.14-1-default (ttyS0).

SSH host key: SHA256:r2d0xtLtSNQOPt0VhMBodwl05cjT9Gcq1of80ZVLUHo (DSA)
SSH host key: SHA256:Y031XUds/7W74LT9hY6oO9LAtIbv118o6tRPvXF486w (ECDSA)
SSH host key: SHA256:HrjLbuj6HDWGvevGCSaG92q1MSTBlXk6AcWsvJQLExA (ED25519)
SSH host key: SHA256:GxJZDfGFv71X20sDKPg36fuJRDyEZGJiBVDQxXG1eX8 (RSA)
ens4:  fe80::5054:ff:fe12:38


localhost login: 


Welcome to openSUSE MicroOS (x86_64) - Kernel 5.16.14-1-default (ttyS0).

SSH host key: SHA256:r2d0xtLtSNQOPt0VhMBodwl05cjT9Gcq1of80ZVLUHo (DSA)
SSH host key: SHA256:Y031XUds/7W74LT9hY6oO9LAtIbv118o6tRPvXF486w (ECDSA)
SSH host key: SHA256:HrjLbuj6HDWGvevGCSaG92q1MSTBlXk6AcWsvJQLExA (ED25519)
SSH host key: SHA256:GxJZDfGFv71X20sDKPg36fuJRDyEZGJiBVDQxXG1eX8 (RSA)
ens4: 10.0.2.16 fe80::5054:ff:fe12:38


localhost login: YpuUt-0-

The log message without IPv4 address is also visible in screenshots after the failing curl invocation (https://openqa.opensuse.org/tests/2246907#step/textinfo/16) and the log message with IPv4 address only appears shortly later (https://openqa.opensuse.org/tests/2246907#step/textinfo/18).

So I would conclude that tests should simply wait until the network is ready (including IPv4) before using it. Hence I'm moving the ticket back to openQA tests.

@mkittler you should be aware that removing the target version and not assigning the ticket to a team (with an according keyword in the subject) while keeping a "[" in the subject will likely lead to the ticket being overlooked for long, see https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging . You gave good suggestions what could be done except that the "networking" test module is microos specific, the test module "textinfo" is more generic, so moving one after the other is not something that works in all cases.

@acarvajal assigning to you as test module maintainer. Could you please update the ticket with the right keyword for "diamond assurance"?

These tickets are not on high pro

okurz wrote:

@mkittler you should be aware that removing the target version and not assigning the ticket to a team (with an according keyword in the subject) while keeping a "[" in the subject will likely lead to the ticket being overlooked for long, see https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging . You gave good suggestions what could be done except that the "networking" test module is microos specific, the test module "textinfo" is more generic, so moving one after the other is not something that works in all cases.

I see in the latest results that this module has not been failing in the linked scenario for at least 3 months, and the last change on the module predates even that date (commit bc26238d4db8ccde9700f6c86c700d32108dd629 from Wed Oct 20 10:00:37 2021 +0200).

I would propose closing this ticket for the time being, and re-opening/creating a new ticket if the issue appears again. As it is, there is nothing to debug or fix FMPOV.

@acarvajal assigning to you as test module maintainer. Could you please update the ticket with the right keyword for "diamond assurance"?

I added the keyword earlier today, but I am thinking on removing it since this is not a module that was used in Diamond Assurance/Workload Testing. In fact, as I told @mkittler via chat, the only reason I appear as maintainer is that there was no maintainer for the module back in 2017 when I adapted it to work on SLES for SAP, and I added myself: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3655#discussion_r143339417

Unless any of you guys have any objection, I will reject & close the ticket next week.

okurz wrote:

acarvajal wrote:

I added the keyword earlier today, but I am thinking on removing it since this is not a module that was used in Diamond Assurance/Workload Testing. In fact, as I told @mkittler via chat, the only reason I appear as maintainer is that there was no maintainer for the module back in 2017 when I adapted it to work on SLES for SAP, and I added myself: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3655#discussion_r143339417

Well, if you don't maintain test modules please make sure there is another better maintainer listed then so that we don't bother you again in the future with requests to you :)

Was this really necessary? Is the smiley at the end added to "soften the blow", because you are aware that this was not necessary? Does it help in solving the issue at hand? Does it add context information? I am aware my comment above does not help in solving the issue, but at least it adds context.

I really want to know, as IMHO this was not necessary or called for. And it is not the first time I have read similar comments from the same source.

So to reply:

• "if you don't maintain test modules" -> Who said I did not?
• "so that we don't bother you again in the future with requests to you" -> Who said this was bothering me?

Just to be clear: I am not bothered by requests, and I can maintain this and other modules in the repo. I am however bothered by the passive-aggressiveness with which you reply to poo's comments any time anybody adds one to a ticket.

So let's cut to the chase, and tell me exactly what you wish me to do:

• Submit a NOOP PR "fixing" a module that for all intents and purposes is working, and link it here to give the impression that the issue was fixed? After all, the ticket was assigned to the module maintainer, suggesting that there is an issue with the module itself, when apparently that is not the actual root cause of the issue as per the analysis in https://progress.opensuse.org/issues/108464#note-8.
• Step into another squad's test and rearrange the order of the test modules as suggested by @mkittler even though that goes beyond the scope of a test module maintainer?
• Submit a PR assigning you as the module maintainer?
• Another suggestion?

Regarding the keyword "[qe-workload-testing]", may I add this to https://progress.opensuse.org/projects/suseqa/ then?

I guess the actual name is SAP Workload Testing: https://confluence.suse.com/display/qasle/QE+squads+-+structure

Unless any of you guys have any objection, I will reject & close the ticket next week.

Well, thank you for checking on the history of job results. I can confirm that it seems the situation has somewhat improved and the test has not failed since some weeks but the observation from mkittler https://progress.opensuse.org/issues/108464#note-8 still applies and I would like someone to do something about this to not waste the brain cycles that have been invested so far. As a last resort we would take over the ticket as QE Tools team again. Please look into the above mentioned suggestions though before assiging to us or me.

Objection noted.

Changing the order of the test modules in the test as suggested in https://progress.opensuse.org/issues/108464#note-8 goes beyond the role of module maintainer AFAIK. Assigning this to @JERiveraMoya and clearing the due date. Joaquin: I hope that's OK; let me know if I should instead assign this to someone else.

acarvajal wrote:

okurz wrote:

acarvajal wrote:

I added the keyword earlier today, but I am thinking on removing it since this is not a module that was used in Diamond Assurance/Workload Testing. In fact, as I told @mkittler via chat, the only reason I appear as maintainer is that there was no maintainer for the module back in 2017 when I adapted it to work on SLES for SAP, and I added myself: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3655#discussion_r143339417

Well, if you don't maintain test modules please make sure there is another better maintainer listed then so that we don't bother you again in the future with requests to you :)

Was this really necessary? Is the smiley at the end added to "soften the blow", because you are aware that this was not necessary? Does it help in solving the issue at hand? Does it add context information? I am aware my comment above does not help in solving the issue, but at least it adds context.

I really want to know, as IMHO this was not necessary or called for. And it is not the first time I have read similar comments from the same source.

So to reply:

• "if you don't maintain test modules" -> Who said I did not?

I am really sorry. By no mean did I want to insult or annoy you. I simply thought that you might not want to continue the maintainership especially also because you removed the team keyword from the subject line.

• "so that we don't bother you again in the future with requests to you" -> Who said this was bothering me?

sorry, seems I misread and misunderstood your earlier comments.

Just to be clear: I am not bothered by requests, and I can maintain this and other modules in the repo. I am however bothered by the passive-aggressiveness with which you reply to poo's comments any time anybody adds one to a ticket.

Sorry, that you feel this way. I will keep this in mind and try to improve but please let me assure you that I had no such intent.

So let's cut to the chase, and tell me exactly what you wish me to do:

• Submit a NOOP PR "fixing" a module that for all intents and purposes is working, and link it here to give the impression that the issue was fixed? After all, the ticket was assigned to the module maintainer, suggesting that there is an issue with the module itself, when apparently that is not the actual root cause of the issue as per the analysis in https://progress.opensuse.org/issues/108464#note-8.

Well, maybe it would actually help if the module checks its prerequisities. But you are right that the actual fix preventing such problem needs to happen elsewhere.

• Step into another squad's test and rearrange the order of the test modules as suggested by @mkittler even though that goes beyond the scope of a test module maintainer?

Well, honestly I think anyone should be welcome to propose improvements, e.g. propose a code change or even directly a pull request and ask the people responsible for a certain area to give feedback. Sure, you don't have to do that as test module maintainer. But often it's easier to propose code changes yourself instead of trying to explain in English rather than code to someone else what would need to be done.

• Submit a PR assigning you as the module maintainer?

I don't think I can be a good module maintainer here

• Another suggestion?

You already picked a good choice with assigning to JERiveraMoya as representative of [y] so adding the according keyword.

Regarding the keyword "[qe-workload-testing]", may I add this to https://progress.opensuse.org/projects/suseqa/ then?

I guess the actual name is SAP Workload Testing: https://confluence.suse.com/display/qasle/QE+squads+-+structure

Now I am not sure what should be added on https://progress.opensuse.org/projects/suseqa/. ticket queries there use the character range [a-z-] so "[qe-workload-testing]" or "[sap-workload-testing]" would work but maybe just "[sap-workload]" would suffice?

Unless any of you guys have any objection, I will reject & close the ticket next week.

Well, thank you for checking on the history of job results. I can confirm that it seems the situation has somewhat improved and the test has not failed since some weeks but the observation from mkittler https://progress.opensuse.org/issues/108464#note-8 still applies and I would like someone to do something about this to not waste the brain cycles that have been invested so far. As a last resort we would take over the ticket as QE Tools team again. Please look into the above mentioned suggestions though before assiging to us or me.

Objection noted.

Changing the order of the test modules in the test as suggested in https://progress.opensuse.org/issues/108464#note-8 goes beyond the role of module maintainer AFAIK. Assigning this to @JERiveraMoya and clearing the due date. Joaquin: I hope that's OK; let me know if I should instead assign this to someone else.