QA (public) » openQA Project (public)

Support for multiple test distributions, just rely on "distri" being specified for now ... in tests_overview and tests is possible to query without specific distri and get results for multiple distri set.

standard grep over testdirectory is too slow .. maybe we can use something like The Silver Searcher / Ripgrep ?

osukup wrote:

Support for multiple test distributions, just rely on "distri" being specified for now
in tests_overview and tests is possible to query without specific distri and get results for multiple distri set.

Not sure if that is a guess or based on reading the code. I'd say when writing tests for now you should assume distri is set. And see later if it's trivial to make more generic.

standard grep over testdirectory is too slow .. maybe we can use something like The Silver Searcher / Ripgrep ?

rg could be faster than grep because it skips a lot in a big test folder, but it would also be a new dep. And it's out of scope for this ticket anyway.

cdywan wrote:

osukup wrote:

Support for multiple test distributions, just rely on "distri" being specified for now
in tests_overview and tests is possible to query without specific distri and get results for multiple distri set.

Not sure if that is a guess or based on reading the code. I'd say when writing tests for now you should assume distri is set. And see later if it's trivial to make more generic.

based on reading code :D , It's much harder made it working with distri than over testdir.

standard grep over testdirectory is too slow .. maybe we can use something like The Silver Searcher / Ripgrep ?

rg could be faster than grep because it skips a lot in a big test folder, but it would also be a new dep. And it's out of scope for this ticket anyway.

rg is in most cases one order faster than grep

my Idea ->

1) add a parameter to overview
2) if is used --> grep files in tests directory
3) extract filenames without extension
4) fill these filenames to 'module' search logic search_args{module}
5) profit ...

osukup wrote:

my Idea ->

1) add a parameter to overview
2) if is used --> grep files in tests directory
3) extract filenames without extension
4) fill these filenames to 'module' search logic search_args{module}
5) profit ...

sounds good! In the ticket description we suggested to use git grep -l which should be way faster than grep or ag of course assuming that we can identify the right test distribution. If this is too hard just iterate over all directories, e.g.

for i in $(find /var/lib/openqa/share/tests -maxdepth 1 -type d); do test -d $i/.git && git -C $i grep -l kontact ; done

or a perl equivalent of the same :)

okurz wrote:

osukup wrote:

my Idea ->

1) add a parameter to overview
2) if is used --> grep files in tests directory
3) extract filenames without extension
4) fill these filenames to 'module' search logic search_args{module}
5) profit ...

sounds good! In the ticket description, we suggested to use git grep -l which should be way faster than grep or ag of course assuming that we can identify the right test distribution. If this is too hard just iterate over all directories, e.g.

git grep isn't much faster than standard grep ( his biggest benefit it limits files for search ), on other side ripgrep is really much faster and the command is pretty simple:

my $res = run_cmd_with_log_return_error(['rg', '-l', $regexp, $dir]);

for i in $(find /var/lib/openqa/share/tests -maxdepth 1 -type d); do test -d $i/.git && git -C $i grep -l kontact ; done

or a perl equivalent of the same :)

osukup wrote:

okurz wrote:

osukup wrote:

my Idea ->

1) add a parameter to overview
2) if is used --> grep files in tests directory
3) extract filenames without extension
4) fill these filenames to 'module' search logic search_args{module}
5) profit ...

sounds good! In the ticket description, we suggested to use git grep -l which should be way faster than grep or ag of course assuming that we can identify the right test distribution. If this is too hard just iterate over all directories, e.g.

git grep isn't much faster than standard grep ( his biggest benefit it limits files for search ), on other side ripgrep is really much faster and the command is pretty simple:

my $res = run_cmd_with_log_return_error(['rg', '-l', $regexp, $dir]);

for i in $(find /var/lib/openqa/share/tests -maxdepth 1 -type d); do test -d $i/.git && git -C $i grep -l kontact ; done

or a perl equivalent of the same :)

on other side this is oppurtinity to look at File::Utils etc modules :D

after going through some File:: perl modules I found there isn't module providing something like python's pathlib :(

so -> now to test @okurz oneliner

after testing oneliner with IPC::Run, its impossible use it with run_cmd_with_log_return_error

any solution with git grep -l needs:

1) something traverse testcasedir and find all subdirs with .git folder (there isn't any nice module for this in Perl :( )
2) then run git grep on this dirs, merge results and they use results

on another side -> using ripgrep brings one call of run_cmd.. on testcasedir with results from all subdirs. --> pretty simple implementation on price one new dep (and only one because ripgrep is literary static binary)

https://github.com/os-autoinst/openQA/pull/4141

how about sh -c 'for i in $(find /var/lib/openqa/share/tests -maxdepth 1 -type d); do test -d $i/.git && git -C $i grep -l kontact ; done'?

We already have code searching files and file contents in lib/OpenQA/WebAPI/Controller/API/V1/Search.pm.

cdywan wrote:

We already have code searching files and file contents in lib/OpenQA/WebAPI/Controller/API/V1/Search.pm.

wau, thx.

used in last version of PR

osukup wrote:

merged :D

It seems like it's deployed on o3, but I don't see any results when trying it e.g. https://openqa.opensuse.org/tests/overview?groupid=24&module_re=openqa
Am I using it wrong?

on small instances, it works as intended -> http://quasar.suse.cz/tests/overview?arch=&flavor=&machine=&test=&modules=&module_re=kernel#

osukup wrote:

cdywan wrote:

osukup wrote:

merged :D

It seems like it's deployed on o3, but I don't see any results when trying it e.g. https://openqa.opensuse.org/tests/overview?groupid=24&module_re=openqa
Am I using it wrong?

looks working for me? https://openqa.opensuse.org/tests/overview?arch=&flavor=&machine=&test=&modules=&module_re=openqa&distri=openqa&version=Tumbleweed&build=%3ATW.9034&groupid=24#

It seems like it breaks when no build is specified. https://openqa.opensuse.org/tests/overview?groupid=24&module_re=openqa&build=%3ATW.9034 works. That's probably unrelated then, filed #98258