Wiki¶

Also see https://progress.opensuse.org/projects/openqav3/wiki

test organization on https://openqa.suse.de/¶

job group names¶

Job group names should be consistent and structured for easy (daily) review of the current status¶

template:

<product_group_short_name> <order_nr>.<product_variant>

e.g. "SLE 12 SP1 1.Server". Keep the whitespace for separation consistent, also see https://progress.opensuse.org/issues/9916

Released products should be named with a prefix 'x' to show up late in the overview page¶

This way we can keep track if tests fail even though the product does not produce new builds. This could help us crosscheck tests. E.g. "x-released SLE 12 SP1 1.Server".

lowercase "x" as all our product names start with capital letters. Sorting works regardless (or uppercase first?).

For now we do not retrigger tests on old builds automatically but any test developer may retrigger it manually, e.g. if he suspects the tests broke and he wants to confirm that local changes are not at fault.

needling best practices¶

There are also other locations where "needling best practices" can be found but we should also have the possibility to keep something on the wiki. Feel free to contact me (okurz) and tell me where it should be instead if there is a better place

applying "workaround" needles¶

If a test reveals a product issue of minor importance it can make sense create a needle with the property "workaround" set. This way, if the needle is matched, the test records this as a "soft-fail". To backtrack the product issue and follow on this and eventually delete the workaround needle if the product issue is fixed, the product issue should be recorded in the needle name itself and at best also in the git commit message adding the needle. If test changes are necessary the source code should have a corresponding comment referencing the issue as well as marking start and stop of the test procedure that is necessary for applying the workaround. Example for a needle name: "gdm-workaround-bsc962806-20160125" referencing bsc#962806

keep in mind:
if two needles match the first in alphabetical list wins, if that one is a workaround needle -> softfail (also see https://github.com/os-autoinst/openQA/issues/779)

do not overwrite old needles because old date confuses people¶

With the needle editor a timestamp of the current day is automatically added to new neeedles. When updating a needle, don't overwrite a needle with the old date tag not to confuse people as it will look really weird in the needle editor.

needle indidvidual column entries in tables¶

Problem: Tables might auto-adjust column size based on content. Therefore it is unsafe to create needles covering multiple columns in a row. Failing example: https://openqa.suse.de/tests/441169#step/yast2_snapper/23
Solution: Needles support multiple areas. Use them to needle individual cells in this example.

Definition of DONE/READY¶

Each of the following points has to be fulfilled to regard individual contributions as DONE. Not every step has to be done by the same step. The overall completion is in responsibility of the complete team.

Definition of DONE¶

Also see http://www.allaboutagile.com/definition-of-done-10-point-checklist/ and https://www.scrumalliance.org/community/articles/2008/september/what-is-definition-of-done-%28dod%29

The following definitions are used to ensure development on individual tests has been completed covering all existing different workflows, e.g. covering "hot-fixes" on the productive instance as well as contributions by new contributors with no previous experience and no control over needle generation on productive instances.

• Code changes are made available via a pull request on the github repository
• New tests as individual test modules (i.e. files under tests/): They are loaded in main.pm of sle and/or opensuse
• "make test" works (e.g. automatic travis CI check triggered on each github PR)
• Guidelines for git commits have been followed
• Code has been reviewed (e.g. in the github PR)
• Favored, but depending on criticality/complexity/size: A local verification test has been run, e.g. post link to a local openQA machine or screenshot or logfile
• Code has been merged (either by reviewer or reviewee after 'LGTM' from others)
• Code has been deployed to osd and o3 (automatic git sync every few minutes)
• If new variables are necessary (feature toggles): A test_suite is executing the test, e.g. test_suite is created or variable is added to existing test_suite over web interface configuration on osd and/or o3
• If a new test_suite has been created: The test_suite is added to at least one job_group
• Necessary needles are made available as PR for sle and/or opensuse (depending if executed, see above for 'main.pm') or are created on the productive instance
• At least one successful test run has been observed on osd or o3 and referenced in the corresponding progress item or bugzilla bug report if one exists

Definition of READY for new tests¶

The following points should be considered before a new test is READY to be implemented:

• Either a product bug has been discovered for which there is no automated test in openQA or a FATE request for new features exists
• A test case description exists depicting the prerequisites of the test, the steps to conduct and the expected result
• The impact and applicability for both SLE and openSUSE products has been considered

Test development instances (staging openQA instances)¶

Contributors cannot afford to verify a newly developed test in all scenarios run by o3 or osd, so tests will break sometime. It would be useful to use a machine to run a subset of the scenarios run in the official instance(s) to make sure the new tests can be deployed with some degree of confidence. But: Any "staging openQA instance" would not be able to run everything which is run in production. It just does not scale. So anyway only a subset can be run and there can be always something missing. Also, we don't have the hardware capacity to cover everything twice and also consider SLE plus openSUSE. Our DOD should cover some important steps so that external contributors are motivated to test something locally first. We have a good test review process and it has to be decided by the reviewer if he accepts the risk of a new test with or without a local verification and covering which scenarios. Depending on the contributors it might make sense to setup a staging server with a subset of tests which is used by multiple test developers to share the burden of openQA setup and administration. For example the YaST team has one available: https://wiki.microfocus.net/index.php/YAST/openQA
If you want to follow this model you can watch this talk by Christopher Hofmann from the OSC16 or ask the YaST team for their experiences.