action #10514

[tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)

Added by okurz about 4 years ago. Updated almost 2 years ago.

Status:ResolvedStart date:01/02/2016
Priority:HighDue date:
Assignee:EDiGiacinto% Done:

0%

Category:Feature requests
Target version:Done
Difficulty:
Duration:

Description

user story

As a test organizer I want unique test module names to prevent ambiguity in references to tests

acceptance criteria

  • AC1: there is a check in "os-autoinst" for test module names
  • AC2: "make test" in os-autoinst-distri-opensuse fails with helpful error message when duplicate test module names are found

further notes

As coolo suggests in
https://github.com/os-autoinst/os-autoinst/pull/406/files#r51416668
"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"

Probably also needs rename of current tests in os-autoinst-distri-opensuse

In general we want a module to potentially appear multiple times within a testflow. openQA jobs can load test modules multiple times. However, when a job does this, the web UI does not show all the module executions. It only shows one execution for each module which is overwritten if the module is executed again., so for e.g. this test:

https://openqa.stg.fedoraproject.org/tests/72160

loads and runs _console_wait_login twice, as you can see from os-autoinst.log, but the web UI shows only one of the runs.

Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.

An alternative solution to that problem might be to handle the same module being called multiple times and make it unique automatically, e.g. by adding a number when the module name is put into the webui display as well as the generated url to the module.


Related issues

Duplicated by openQA Project - action #17260: When a job loads module(s) more than once, the web UI onl... Rejected 22/02/2017
Duplicated by openQA Project - action #17628: test-module doesn't appear if it gets triggered twice Rejected 09/03/2017
Precedes openQA Tests - action #28678: Use new way of loading test modules multiple times withou... Resolved 02/02/2016

History

#1 Updated by okurz about 4 years ago

Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.

#2 Updated by okurz about 3 years ago

  • Category set to Feature requests

#3 Updated by AdamWill about 3 years ago

this shouldn't prevent running the same module multiple times, though. AIUI, that's a feature.

#4 Updated by okurz almost 3 years ago

  • Related to action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added

#5 Updated by okurz almost 3 years ago

  • Description updated (diff)

#6 Updated by okurz almost 3 years ago

  • Related to deleted (action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module)

#7 Updated by okurz almost 3 years ago

  • Duplicated by action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added

#8 Updated by okurz almost 3 years ago

  • Duplicated by action #17628: test-module doesn't appear if it gets triggered twice added

#9 Updated by okurz almost 3 years ago

  • Subject changed from Prevent test module name clashes to Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)

#10 Updated by oholecek almost 3 years ago

  • Assignee set to oholecek

There was already some work done on this done by okurz, I have a last piece in my git branch https://github.com/aaannz/openQA/tree/multi-modules but since I didn't touch that for a long time I need to test it and write some tests for it.
I'll take a look at this since it started appearing more often.

#11 Updated by coolo over 2 years ago

  • Assignee deleted (oholecek)
  • Priority changed from Normal to High
  • Target version set to Ready

This needs to be finished by someone else :(

#12 Updated by szarate over 2 years ago

  • Subject changed from Prevent test module name clashes (problem on loading test module twice / multiple times / more than one) to [tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)

This will be part of proposing a solution/investigating (I.E, not too much time to be invested), but following the idea of @coolo to begin with:

"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"    

For the future (If it makes sense): However, having the possibility to have the same module called multiple times within a test run?... why not? :)

#13 Updated by coolo over 2 years ago

isotovideo was already adapted afaik - my comment is old.

The acceptenace criterium is basically:

loadtest "x11/helloworld";
loadtest "console/helloworld";
loadtest "console/setvar_hello";
loadtest "console/helloworld";

should run - and show up 3 helloworld modules in the webui.

#14 Updated by okurz over 2 years ago

Correct. isotovideo already supports this with no problem. Only in the openQA webui the later execution of the same module overwrite the former.

#15 Updated by AdamWill over 2 years ago

Yeah. We were actually running the same test module multiple times in the same job in production in Fedora for a while. os-autoinst copes with it fine, the test runs as expected. But it has various weird consequences in the webUI. So for now, we're working around this with some duct tape (symlinks of the modules we want to run more than once, and a stupid trick in main.pm for loading them):

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/main.pm#_193-238

#16 Updated by okurz over 2 years ago

We have the same approach for SLE/openSUSE tests, symlink modules. Works so good I would consider loading the exact same module as unimportant and I would not mind if we prevent that from os-autoinst and openQA with a fatal die

#17 Updated by coolo over 2 years ago

I do mind - as it makes dynamic solutions for main.pm impossible

#18 Updated by coolo over 2 years ago

#20 Updated by EDiGiacinto over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to EDiGiacinto

PR: https://github.com/os-autoinst/os-autoinst/pull/884 - needs more testing/unit tests still

#21 Updated by EDiGiacinto about 2 years ago

  • Status changed from In Progress to Feedback

#22 Updated by EDiGiacinto about 2 years ago

  • Status changed from Feedback to Resolved

Criteria should have been met in https://github.com/os-autoinst/os-autoinst/pull/884 and has been merged, please re-open if necessary.

#23 Updated by okurz about 2 years ago

  • Precedes action #28678: Use new way of loading test modules multiple times without file duplication added

#24 Updated by szarate almost 2 years ago

  • Target version changed from Ready to Done

Also available in: Atom PDF