Project

General

Profile

action #34486

database of "test cases" or how to search for tests we have in openQA

Added by okurz almost 3 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2018-04-08
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

From time to time a request comes up "how to search for test cases in openQA". In many cases the definition of "test cases" is not clear to the person asking or no clear relation to our openQA terms exists. However, we should be able to provide an easy search interface which will simply by default return search results matching any of "testsuites", "scenarios", "test modules", "test steps".

Acceptance criteria

  • DONE AC1: There is a common way to search for "tests" in openQA as a catch-all however ambiguous the term is understood
  • DONE AC2: The user is asked one and only one question, that is the search term, nothing else
  • AC3: Given a generic search term, When that search term matches any of test suite, scenario, test module, test steps or just mention in test code, Then all definition places are shown

Acceptance tests

  • DONE AT1-1: manual UX test: An uninformed user can find the search field over the openQA web UI
  • AT3-1: search term "foo" finds a test suite "foo"
  • AT3-2: search term "foo" finds a test suite "123foo456"
  • DONE AT3-3: search term "foo" finds a scenario "openSUSE-textmode@foo" for machine "foo"
  • DONE AT3-4: search term "foo" finds a test module "tests/console/foo.pm" from a test distribution folder within /var/lib/openqa/share/tests/*/
  • DONE AT3-5: search term "foo" finds a reference to "foo" in code within a test module "tests/console/bar.pm" as in AT3-4

Suggestion


Related issues

Related to openQA Project - action #38024: [functional][y] Add ability to search for a module occurrences in all the tests of job groupResolved2018-06-292019-01-29

Has duplicate openQA Project - action #65954: Create a way to check which jobs contain a testRejected2020-04-21

Copied to openQA Project - coordination #70840: [epic] further features for the "search" featureNew2020-09-16

History

#1 Updated by szarate almost 3 years ago

  • Description updated (diff)

okurz by:

**AC2:** The user is one and only one question, that is the search term, nothing else    

You mean that the user has one and only one question? :).

I guess that for things like this, things would need to be indexed, and that filtering (eg. by build or distri) would be only for names of test suites and scenarios.

#2 Updated by okurz almost 3 years ago

szarate wrote:

okurz by:

**AC2:** The user is one and only one question, that is the search term, nothing else    

You mean that the user has one and only one question? :).

I am assuming that the user does not know if he means a testsuite, test module, test step, scenario, product, etc. So I envision the default to be simplistic, like google once was, just a single text field. Ok, you can provide buttons below or above the search bar for advanced stuff, help, customization but let's start simple :)

I guess that for things like this, things would need to be indexed, and that filtering (eg. by build or distri) would be only for names of test suites and scenarios.

What do you want to index? We have git repos -> already indexed, and we have database tables for test suites, so go with that :)

#3 Updated by EDiGiacinto almost 3 years ago

okurz wrote:

szarate wrote:

okurz by:

**AC2:** The user is one and only one question, that is the search term, nothing else    

You mean that the user has one and only one question? :).

I am assuming that the user does not know if he means a testsuite, test module, test step, scenario, product, etc. So I envision the default to be simplistic, like google once was, just a single text field. Ok, you can provide buttons below or above the search bar for advanced stuff, help, customization but let's start simple :)

While i can agree with being simplistic i don't share re-implementing google.

I guess that for things like this, things would need to be indexed, and that filtering (eg. by build or distri) would be only for names of test suites and scenarios.

What do you want to index? We have git repos -> already indexed, and we have database tables for test suites, so go with that :)

What you propose basically is a killer (as in resources) api endpoint that does a complex query - including search in a git subtree. That's why you might want an index where to search data instead of doing the search directly whenever a user request it.

#4 Updated by rpalethorpe almost 3 years ago

Oliver just try screen scraping the whole of OpenQA and index every page with Xapian (see https://gitlab.suse.de/alnovak/fast-bzsearch/tree/master). As you are basically asking for a totally unstructured text search which OpenQA has no infrastructure to support, Xapian is really fast and you already know how to screen scrape OpenQA.

#5 Updated by okurz over 2 years ago

  • Description updated (diff)

#6 Updated by okurz about 2 years ago

  • Related to action #38024: [functional][y] Add ability to search for a module occurrences in all the tests of job group added

#7 Updated by okurz 9 months ago

  • Has duplicate action #65954: Create a way to check which jobs contain a test added

#8 Updated by okurz 9 months ago

  • Description updated (diff)

#10 Updated by mkittler 8 months ago

I'm really wondering about the idea to search the file system. Beside the problem pointed out by @EDiGiacinto I'm wondering what you want to do once you've found a file in the repository. There's no relation from such a file to a concrete openQA job. But openQA jobs are what the results of that search are supposed to be, right? The acceptance criteria should clarify what kind of results the search is supposed to reveal. I assume visitors of the openQA page would expect to find concrete openQA jobs and if they wanted to search for particular files within a certain test distribution they would use a different kind of search in the first place.

#11 Updated by cdywan 8 months ago

mkittler wrote:

I'm really wondering about the idea to search the file system. Beside the problem pointed out by @EDiGiacinto I'm wondering what you want to do once you've found a file in the repository. There's no relation from such a file to a concrete openQA job. But openQA jobs are what the results of that search are supposed to be, right? The acceptance criteria should clarify what kind of results the search is supposed to reveal. I assume visitors of the openQA page would expect to find concrete openQA jobs and if they wanted to search for particular files within a certain test distribution they would use a different kind of search in the first place.

Not sure where searching the file system comes from. But one of the duplicates was based on this real use case:

  • Allison Average looks at #63244.
  • The example URLs are dead.
  • So she wants to find jobs using qa_run.

#12 Updated by okurz 8 months ago

  • Description updated (diff)

mkittler wrote:

[…] There's no relation from such a file to a concrete openQA job

Correct. But imagine the following: A QA automation engineer wonders if any test coverage for nginx exists. So you might call https://openqa.opensuse.org/tests?match=nginx and find nothing because there is no test suite or scenario that mentions "nginx" in the name.

But openQA jobs are what the results of that search are supposed to be, right?

No. As above, there might be no test suite or scenario that mention "nginx", but the openqa test suite "textmode" might include a test module from os-autoinst-distri-opensuse called "httpd.pm" that has zypper -n in nginx && systemctl start nginx. You can only find that by looking into the test distribution code, i.e. files on the file system or the mentioned search in the repo using the github search.

The acceptance criteria should clarify what kind of results the search is supposed to reveal. I assume visitors of the openQA page would expect to find concrete openQA jobs and if they wanted to search for particular files within a certain test distribution they would use a different kind of search in the first place.

My experience tells me that they don't. It happened multiple times that QA engineers create a pull request – after weeks worth of work invested! – to add test code for something that already exists and they were not aware. Why? Mainly because they never searched simply in os-autoinst-distri-opensuse for mentions of the package/component/file/service/module/feature to be tested.

I do not see the relation that cdywan has mentioned relating to #63244 but the other related issues linked to this ticket here already show other examples of the same.

I think the acceptance criteria are not vague but generic and they should be. Still I tried to extend them now also with acceptance tests to show even more explicitly what I expect

#13 Updated by okurz 8 months ago

  • Description updated (diff)
  • Status changed from New to Workable

Add more suggestions. If you think like this the ticket is feasible and well-specified but too big we can split it in sub tasks, e.g. "generic search API route /api/v1/search looking at test suites only (as a first step)"

#14 Updated by mkittler 7 months ago

Ok, so this ticket is about combining different places to search but not being smart about the relations of the things we might find. So if "nginx" is found in a file of the test distribution that file's path is part of the results but there are still no related openQA jobs returned unless "nginx" shows up within the job settings or job module names.

#15 Updated by okurz 7 months ago

yes

#16 Updated by cdywan 7 months ago

  • Assignee set to cdywan

#17 Updated by cdywan 7 months ago

  • Status changed from Workable to In Progress

After having researched what's needed for this, I took a step back and pushed code for the following aspects:

  • Search results view
  • AT3-4 - Search for Perl modules by filename
  • AT3-5 - Search Perl module contents

https://github.com/os-autoinst/openQA/pull/3237

#18 Updated by okurz 6 months ago

  • Target version set to Ready

#19 Updated by cdywan 5 months ago

The initial implementation of the global search UI as well as the underlying REST API got merged - we decided to put it in the experimental namespace for now.

As per discussion, I'm also proposing a follow-up to improve general responsiveness: https://github.com/os-autoinst/openQA/pull/3313

#20 Updated by mgrifalconi 5 months ago

Hello, I would like to propose the following use case.

  • I am working with mau-extratests test suite
  • I know which perl modules it runs
  • I don't know how to find openqa tests to clone that run mau-extratests

  • I search for mau-extratests (possibly filtering only for job groups results)

  • I get as result, between the others:

The job template: https://openqa.suse.de/admin/job_templates/232
because it contains:

- mau-extratests
- mau-filesystem
[..]
- mau-extratests-desktop:

Thanks!

#21 Updated by cdywan 5 months ago

#22 Updated by cdywan 5 months ago

mgrifalconi wrote:

Hello, I would like to propose the following use case.

  • I am working with mau-extratests test suite
  • I know which perl modules it runs
  • I don't know how to find openqa tests to clone that run mau-extratests
  • I search for mau-extratests (possibly filtering only for job groups results)
  • I get as result, between the others:

The job template: https://openqa.suse.de/admin/job_templates/232
because it contains:

- mau-extratests
- mau-filesystem
[..]
- mau-extratests-desktop:

Thank you for your feedback! I prepared a branch that searches job templates, which I think is what you want:

https://github.com/os-autoinst/openQA/pull/3320

#23 Updated by cdywan 5 months ago

  • Description updated (diff)

#24 Updated by cdywan 5 months ago

More potentially interesting match types:

  • Needles
  • Assets
  • Bugs
  • Job by job module

  • I will also propose a PR to handle the testrepo symlink to avoid redundant results.

  • mkittler also suggested we display the type of result, maybe as an icon.

  • We should link results, e.g. a job template links to the job group.

#25 Updated by cdywan 5 months ago

cdywan wrote:

#26 Updated by okurz 5 months ago

As discussed in the weekly QA tools team meeting a next idea would be to return current job module results similar to what https://github.com/okurz/scripts/blob/master/openqa-db_query_last_use_of_module as mentioned in the description already provides.

#27 Updated by okurz 5 months ago

#28 Updated by okurz 5 months ago

I moved out all further, additional ideas into #70840 , please focus on adding the "job module results" only today to be able to resolve this ticket.

#30 Updated by cdywan 4 months ago

  • Status changed from In Progress to Feedback

#31 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

so https://github.com/os-autoinst/openQA/pull/3353 is in and also deployed on osd now. Searching for job modules also works now, e.g. https://openqa.opensuse.org/search?q=kontact

All the rest is left for #70840

Also available in: Atom PDF