QA » openQA Project » openQA Tests

This ticket should help with preparing of the automated deployments or rather the triggering of new Service Packs into Development or Maintenance phases, for this we need a new epic.

This epic should stay only for static checks and code improvements.

• automatic push of use strict; warnings and testapi goes in here too.

From: https://suse.slack.com/archives/C02CANHLANP/p1673359533046859?thread_ts=1673351658.066219&cid=C02CANHLANP

I always ended up calling the stuff directly on my own and never bothered to go further, but having make targets that allow parameters makes so much more sense...
and having a target similar to make tidy, we can actually do the same with make tests so the behavior only affects modified files.

From our discussion in RC linked in previous: It'd be great to have way to easily run all tests which are in CI (./tools/check_metadata, ./tools/tidy, ...) on single file or list of specified files. What I usually need is to quickly test 1-3 files. Make targets used for testing, which are specified in top level Makefile are run on all files, which does not scale.

There could be ./tools/test-everything, which would expect list of files and run all tests on them (or at least these which can be run easily locally). The problem is that some commands are defined in the top level Makefile (e.g. perlcritic), all of these would have to be also put as scripts in ./tools/. Having commands defined in Makefile can brings complications (at least you need to have make installed, I'd try to have minimum scripting in Makefile, just call scripts in ./tools/).

Not only I want fast way to test everything I modified locally. Other problem, which bothers me is the long output, when everything is tested with pipes. Pipes does not print affected file, thus I need to scroll to find which tool reported particular error and run it locally on files I modified to see the problem.

Running locally I get files:

./tools/check_metadata $(git ls-files)
Missing Copyright <year> SUSE LLC in data/ai_ml/tvm/post_processing.py
Missing '# Summary: <multi line summary of test>' in data/apparmor/adminer.php5
Missing '# Maintainer: <email address>' in data/apparmor/adminer.php5

But in CI I see many useless output, which makes things hard to find:

tools/check_metadata $(git ls-files "tests/**.pm")
# Maintainer: Guillaume GARDET <guillaume@opensuse.org>
# Maintainer: Guillaume GARDET <guillaume@opensuse.org>
# Maintainer: QE Security <none@suse.de>`

The same problem is with other tests perlcritic.

Adding a quick note here of things that are needed:

• A checker that intends to guess/understand whether a test really needs to use a tty or if it can use serial-terminal instead