Project

General

Profile

Actions

action #10514

closed

[tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)

Added by okurz almost 9 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2016-02-01
Due date:
% Done:

0%

Estimated time:

Description

user story

As a test organizer I want unique test module names to prevent ambiguity in references to tests

acceptance criteria

  • AC1: there is a check in "os-autoinst" for test module names
  • AC2: "make test" in os-autoinst-distri-opensuse fails with helpful error message when duplicate test module names are found

further notes

As coolo suggests in
https://github.com/os-autoinst/os-autoinst/pull/406/files#r51416668
"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"

Probably also needs rename of current tests in os-autoinst-distri-opensuse

In general we want a module to potentially appear multiple times within a testflow. openQA jobs can load test modules multiple times. However, when a job does this, the web UI does not show all the module executions. It only shows one execution for each module which is overwritten if the module is executed again., so for e.g. this test:

https://openqa.stg.fedoraproject.org/tests/72160

loads and runs _console_wait_login twice, as you can see from os-autoinst.log, but the web UI shows only one of the runs.

Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.

An alternative solution to that problem might be to handle the same module being called multiple times and make it unique automatically, e.g. by adding a number when the module name is put into the webui display as well as the generated url to the module.


Related issues 3 (0 open3 closed)

Has duplicate openQA Project - action #17260: When a job loads module(s) more than once, the web UI only shows one run of each moduleRejected2017-02-22

Actions
Has duplicate openQA Project - action #17628: test-module doesn't appear if it gets triggered twiceRejected2017-03-09

Actions
Precedes openQA Tests - action #28678: Use new way of loading test modules multiple times without file duplicationResolvedokurz2016-02-02

Actions
Actions #1

Updated by okurz almost 9 years ago

Although right now we have the same module name in different components we should try to avoid it as the URL does not differ, i.e. https://openqa.suse.de/tests/195580/modules/shutdown refers to "x11-shutdown" but the URL name is ambiguous.

Actions #2

Updated by okurz almost 8 years ago

  • Category set to Feature requests
Actions #3

Updated by AdamWill over 7 years ago

this shouldn't prevent running the same module multiple times, though. AIUI, that's a feature.

Actions #4

Updated by okurz over 7 years ago

  • Related to action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added
Actions #5

Updated by okurz over 7 years ago

  • Description updated (diff)
Actions #6

Updated by okurz over 7 years ago

  • Related to deleted (action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module)
Actions #7

Updated by okurz over 7 years ago

  • Has duplicate action #17260: When a job loads module(s) more than once, the web UI only shows one run of each module added
Actions #8

Updated by okurz over 7 years ago

  • Has duplicate action #17628: test-module doesn't appear if it gets triggered twice added
Actions #9

Updated by okurz over 7 years ago

  • Subject changed from Prevent test module name clashes to Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)
Actions #10

Updated by oholecek over 7 years ago

  • Assignee set to oholecek

There was already some work done on this done by okurz, I have a last piece in my git branch https://github.com/aaannz/openQA/tree/multi-modules but since I didn't touch that for a long time I need to test it and write some tests for it.
I'll take a look at this since it started appearing more often.

Actions #11

Updated by coolo almost 7 years ago

  • Assignee deleted (oholecek)
  • Priority changed from Normal to High
  • Target version set to Ready

This needs to be finished by someone else :(

Actions #12

Updated by szarate almost 7 years ago

  • Subject changed from Prevent test module name clashes (problem on loading test module twice / multiple times / more than one) to [tools][sprint 201711.2] Prevent test module name clashes (problem on loading test module twice / multiple times / more than one)

This will be part of proposing a solution/investigating (I.E, not too much time to be invested), but following the idea of @coolo to begin with:

"IMO loadtest should fail - right now it stores the tests in %tests by $fullname. I would make that fatal by $name"    

For the future (If it makes sense): However, having the possibility to have the same module called multiple times within a test run?... why not? :)

Actions #13

Updated by coolo almost 7 years ago

isotovideo was already adapted afaik - my comment is old.

The acceptenace criterium is basically:

loadtest "x11/helloworld";
loadtest "console/helloworld";
loadtest "console/setvar_hello";
loadtest "console/helloworld";

should run - and show up 3 helloworld modules in the webui.

Actions #14

Updated by okurz almost 7 years ago

Correct. isotovideo already supports this with no problem. Only in the openQA webui the later execution of the same module overwrite the former.

Actions #15

Updated by AdamWill almost 7 years ago

Yeah. We were actually running the same test module multiple times in the same job in production in Fedora for a while. os-autoinst copes with it fine, the test runs as expected. But it has various weird consequences in the webUI. So for now, we're working around this with some duct tape (symlinks of the modules we want to run more than once, and a stupid trick in main.pm for loading them):

https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/main.pm#_193-238

Actions #16

Updated by okurz almost 7 years ago

We have the same approach for SLE/openSUSE tests, symlink modules. Works so good I would consider loading the exact same module as unimportant and I would not mind if we prevent that from os-autoinst and openQA with a fatal die

Actions #17

Updated by coolo almost 7 years ago

I do mind - as it makes dynamic solutions for main.pm impossible

Actions #18

Updated by coolo almost 7 years ago

Actions #20

Updated by EDiGiacinto almost 7 years ago

  • Status changed from New to In Progress
  • Assignee set to EDiGiacinto

PR: https://github.com/os-autoinst/os-autoinst/pull/884 - needs more testing/unit tests still

Actions #21

Updated by EDiGiacinto almost 7 years ago

  • Status changed from In Progress to Feedback
Actions #22

Updated by EDiGiacinto almost 7 years ago

  • Status changed from Feedback to Resolved

Criteria should have been met in https://github.com/os-autoinst/os-autoinst/pull/884 and has been merged, please re-open if necessary.

Actions #23

Updated by okurz almost 7 years ago

  • Precedes action #28678: Use new way of loading test modules multiple times without file duplication added
Actions #24

Updated by szarate over 6 years ago

  • Target version changed from Ready to Done
Actions

Also available in: Atom PDF