action #10514
closed[tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)
0%
Description
user story¶
As a test organizer I want unique test module names to prevent ambiguity in references to tests
acceptance criteria¶
- AC1: there is a check in "os-autoinst" for test module names
- AC2: "make test" in os-autoinst-distri-opensuse fails with helpful error message when duplicate test module names are found
further notes¶
As coolo suggests in
https://github.com/os-autoinst/os-autoinst/pull/406/files#r51416668
"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"
Probably also needs rename of current tests in os-autoinst-distri-opensuse
In general we want a module to potentially appear multiple times within a testflow. openQA jobs can load test modules multiple times. However, when a job does this, the web UI does not show all the module executions. It only shows one execution for each module which is overwritten if the module is executed again., so for e.g. this test:
https://openqa.stg.fedoraproject.org/tests/72160
loads and runs _console_wait_login
twice, as you can see from os-autoinst.log
, but the web UI shows only one of the runs.
Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.
An alternative solution to that problem might be to handle the same module being called multiple times and make it unique automatically, e.g. by adding a number when the module name is put into the webui display as well as the generated url to the module.
Updated by okurz almost 9 years ago
Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.
Updated by AdamWill over 7 years ago
this shouldn't prevent running the same module multiple times, though. AIUI, that's a feature.
Updated by okurz over 7 years ago
- Related to action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added
Updated by okurz over 7 years ago
- Related to deleted (action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module)
Updated by okurz over 7 years ago
- Has duplicate action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added
Updated by okurz over 7 years ago
- Has duplicate action #17628: test-module doesn't appear if it gets triggered twice added
Updated by okurz over 7 years ago
- Subject changed from Prevent test module name clashes to Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)
Updated by oholecek over 7 years ago
- Assignee set to oholecek
There was already some work done on this done by okurz, I have a last piece in my git branch https://github.com/aaannz/openQA/tree/multi-modules but since I didn't touch that for a long time I need to test it and write some tests for it.
I'll take a look at this since it started appearing more often.
Updated by coolo almost 7 years ago
- Assignee deleted (
oholecek) - Priority changed from Normal to High
- Target version set to Ready
This needs to be finished by someone else :(
Updated by szarate almost 7 years ago
- Subject changed from Prevent test module name clashes (problem on loading test module twice / multiple times / more than one) to [tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)
This will be part of proposing a solution/investigating (I.E, not too much time to be invested), but following the idea of @coolo to begin with:
"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"
For the future (If it makes sense): However, having the possibility to have the same module called multiple times within a test run?... why not? :)
Updated by coolo almost 7 years ago
isotovideo was already adapted afaik - my comment is old.
The acceptenace criterium is basically:
loadtest "x11/helloworld";
loadtest "console/helloworld";
loadtest "console/setvar_hello";
loadtest "console/helloworld";
should run - and show up 3 helloworld modules in the webui.
Updated by okurz almost 7 years ago
Correct. isotovideo already supports this with no problem. Only in the openQA webui the later execution of the same module overwrite the former.
Updated by AdamWill almost 7 years ago
Yeah. We were actually running the same test module multiple times in the same job in production in Fedora for a while. os-autoinst copes with it fine, the test runs as expected. But it has various weird consequences in the webUI. So for now, we're working around this with some duct tape (symlinks of the modules we want to run more than once, and a stupid trick in main.pm
for loading them):
https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/main.pm#_193-238
Updated by okurz almost 7 years ago
We have the same approach for SLE/openSUSE tests, symlink modules. Works so good I would consider loading the exact same module as unimportant and I would not mind if we prevent that from os-autoinst and openQA with a fatal die
Updated by coolo almost 7 years ago
I do mind - as it makes dynamic solutions for main.pm impossible
Updated by coolo almost 7 years ago
See https://progress.opensuse.org/issues/27124#note-1 for yet another use case
Updated by EDiGiacinto almost 7 years ago
- Status changed from New to In Progress
- Assignee set to EDiGiacinto
PR: https://github.com/os-autoinst/os-autoinst/pull/884 - needs more testing/unit tests still
Updated by EDiGiacinto almost 7 years ago
- Status changed from In Progress to Feedback
Updated by EDiGiacinto almost 7 years ago
- Status changed from Feedback to Resolved
Criteria should have been met in https://github.com/os-autoinst/os-autoinst/pull/884 and has been merged, please re-open if necessary.
Updated by okurz almost 7 years ago
- Precedes action #28678: Use new way of loading test modules multiple times without file duplication added