Introduction¶

This is the organisation wiki for the openQA Project.
The source code is hosted in the os-autoinst github project, especially openQA itself and the main backend os-autoinst

If you are interested in the tests for SUSE/openSUSE products take a look into the openqatests project.

• Table of contents
• Introduction
• Organisational
  • ticket workflow
  • openQA calls
    • "openQA backend coordination" call
    • "SUSE QA test coordination" call
  • ticket templates
    • defects
    • feature requests
  • pull request handling on github
  • Where to contribute?
• Use cases
  • Use case 1
  • Use case 2
  • Use case 3
  • Use case 4
  • Use case 5
  • Use case 6
  • Use case 7
• Glossary
• Thoughts about categorizing test results, issues, states within openQA
  • Nomenclature
    • Test status categories
    • Priorization of work regarding categories
  • Further categorization of statuses, issues and such in testing, especially automatic tests
    • Categorization scheme
    • Further decision steps working on test issues
• Advanced features in openQA
  • Show previous results in test results page gh#538
  • Link to latest in scenario name gh#836
  • Add 'latest' query route gh#815
  • Add web UI controls to select 20/50/100/400 previous results gh#744
  • Show bug or label icon on overview if labeled gh#550
  • Show certificate next to builds on overview if all failures are labeled gh#560, gh#1052
  • Allow group overview query by result gh#531
  • Add web UI controls to select more builds in group_overview gh#804
  • Add more query parameters for configuring last builds gh#575
  • Build tagging and keeping important builds gh#591
    • Tag builds with special comments on group overview
    • Keeping important builds
    • Example screenshot of a tag coment and corresponding tagged build
  • Add web UI controls to filter only tagged or all builds gh#807
  • Carry over bugrefs from previous jobs in same scenario if still failing gh#564
  • Distinguish product and test issues bugref gh#708
  • Pinning comments as group description
  • Filtering test results in test result overview
  • Proposals for uses of labels

Organisational¶

ticket workflow¶

The following ticket statuses are used together and their meaning is explained:

• New: No one has worked on the ticket or no one is feeling responsible for the work on this ticket
• In Progress: Any state between New and Resolved
• Resolved: The complete work on this issue is done and the according issue is supposed to be fixed as observed (Should be updated together with a link to a merged pull request or also a link to an production openQA showing the effect)
• Feedback: Further work on the ticket is blocked by external dependency or open points. Sometimes also used to ask Assignee about progress on inactivity
• Rejected: The issue is considered invalid, should not be done, is considered out of scope.
• Closed: As this can be set only by administrators it is suggested to not use this status.

It is good practice to update the status together with a comment about it, e.g. a link to a pull request or a reason for reject.

openQA calls¶

Currently there are two recurring openQA calls conducted at SUSE on http://jangouts.suse.de/. If there would be more interest from the outside the call could be made on a public platform.

Both meetings should target to finish in 15 minutes each. If more time is needed, propose to stay in the call with the required subset of attendees.

Standard rules of good "standup meetings" apply here, too, e.g.

• Be on time (be there at meeting start)
• Be concise (help keep the time limit)
• Be polite
• focus on
  • what you achieved
  • what you plan
  • where did you face problems where you could use help

"openQA backend coordination" call¶

objectives:

• Coordination on openQA backend development

execution: A regular daily call from Mon-Fri at 1000 CET/CEST

"SUSE QA test coordination" call¶

objectives:

• Coordination on openQA based test development, especially SLE products
• Information about important development in openQA backend by backend responsibles

execution: Mon + Wed, at 1030 CET/CEST

If somebody from SUSE QA team will do back-end development he can attend the first call as well, of course.

ticket templates¶

You can use these templates to fill in tickets and further improve them with more detail over time. Copy the code block, paste it into a new issue, replace every block marked with "<…>" with your content or delete if not appropriate.

defects¶

Subject: <Short description, example: "openQA dies when triggering any Windows ME tests">

## observation
<description of what can be observed and what the symptoms are, provide links to failing test results and/or put short blocks from the log output here to visualize what is happening>

## steps to reproduce
* <do this>
* <do that>
* <observe result>

## problem
<problem investigation, can also include different hypotheses, should be labeled as "H1" for first hypothesis, etc.>

## suggestion
<what to do as a first step>

## workaround
<example: retrigger job>

example ticket: #10526

feature requests¶

Subject: <Short description, example: "grub3 btrfs support" (feature)>

## User story
<As a <role>, I want to <do an action>, to <achieve which goal> >

## acceptance criteria
* <**AC1:** the first acceptance criterion that needs to be fulfilled to do this, example: Clicking "restart button" causes restart of the job>
* <**AC2:** also think about the "not-actions", example: other jobs are not affected>

## tasks
* <first task to do as an easy starting point>
* <what do do next>
* <optional: mark "optional" tasks>

## further details
<everything that does not fit into above sections>

example ticket: #10212

pull request handling on github¶

As a reviewer of pull requests on github for all related repositories, e.g. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pulls, apply labels in case PRs are open for a longer time and can not be merged so that we keep our backlog clean and know why PRs are blocked.

• notready: Triaged as not ready yet for merging, no (immediate) reaction by the reviewee, e.g. when tests are missing, other scenarios break, only tested for one of SLE/TW
• wip: Marked by the reviewee itself as "[WIP]" or "[DO-NOT-MERGE]" or similar
• question: Questions to the reviewee, not answered yet

Where to contribute?¶

If you want to help openQA development you can take a look into the existing issues. There are also some "always valid" tasks to be working on:

• improve test coverage:
  • user story: As openqa backend as well as test developer I want better test coverage of our projects to reduce technical debt
  • acceptance criteria: test coverage is significantly higher than before
  • suggestions: check current coverage in each individual project (os-autoinst/openQA/os-autoinst-distri-opensuse) and add tests as necessary

Use cases¶

The following use cases 1-6 have been defined within a SUSE workshop (others have been defined later) to clarify how different actors work with openQA. Some of them are covered already within openQA quite well, some others are stated as motivation for further feature development.

Use case 1¶

User: QA-Project Managment
primary actor: QA Project Manager, QA Team Leads
stakeholder: Directors, VP
trigger: product milestones, providing a daily status
user story: „As a QA project manager I want to check on a daily basis the „openQA Dashboard“ to get a summary/an overall status of the „reviewers results“ in order to take the right actions and prioritize tasks in QA accordingly.“

Use case 2¶

User: openQA-Admin
primary actor: Backend-Team
stakeholder: Qa-Prjmgr, QA-TL, openQA Tech-Lead
trigger: Bugs, features, new testcases
user story: „As an openQA admin I constantly check in the web-UI the system health and I manage its configuration to ensure smooth operation of the tool.“

Use case 3¶

User: QA-Reviewer
primary actor: QA-Team
stakeholder: QA-Prjmgr, Release-Mgmt, openQA-Admin
trigger: every new build
user story: „As an openQA-Reviewer at any point in time I review on the webpage of openQA the overall status of a build in order to track and find bugs, because I want to find bugs as early as possible and report them.“

Use case 4¶

User: Testcase-Contributor
primary actor: All development teams, Maintenance QA
stakeholder: QA-Reviewer, openQA-Admin, openQA Tech-Lead
trigger: features, new functionality, bugs, new product/package
user story: „As developer when there are new features, new functionality, bugs, new product/package in git I contribute my testcases because I want to ensure good quality submissions and smooth product integration.“

Use case 5¶

User: Release-Mgmt
primary actor: Release Manager
stakeholder: Directors, VP, PM, TAMs, Partners
trigger: Milestones
user story: „As a Release-Manager on a daily basis I check on a dashboard for the product health/build status in order to act early in case of failures and have concrete and current reports.“

Use case 6¶

User: Staging-Admin
primary actor: Staging-Manager for the products
stakeholder: Release-Mgmt, Build-Team
trigger: every single submission to projects
user story: „As a Staging-Manager I review the build status of packages with every staged submission to the „staging projects“ in the „staging dashboard“ and the test-status of the pre-integrated fixes, because I want to identify major breakage before integration to the products and provide fast feedback back to the development.“

Use case 7¶

User: Bug investigator
primary actor: Any bug assignee for openQA observed bugs
stakeholder: Developer
trigger: bugs
user story: „As a developer that has been assigned a bug which has been observed in openQA I can review referenced tests, find a newer and the most recent job in the same scenario, understand what changed since the last successful job, what other jobs show same symptoms to investigate the root cause fast and use openQA for verification of a bug fix.“

Glossary¶

The following terms are used within the context of openQA:

• test modules: an individual test case in a single perl module file, e.g. "sshxterm". If not further specified a test module is denoted with its "short name" equivalent to the filename including the test definition. The "full name" is composed of the test group (TBC), which itself is formed by the top-folder of the test module file, and the short name, e.g. "x11-sshxterm" (for x11/sshxterm.pm)
• test suite: a collection of test modules, e.g. "textmode". All test modules within one test suite are run serially
• job: one run of individual test cases in a row denoted by a unique number for one instance of openQA, e.g. one installation with subsequent testing of applications within gnome
• test run: equivalent to job
• test result: the result of one job, e.g. "passed" with the details of each individual test module
• test step: the execution of one test module within a job
• distri: a test distribution but also sometimes referring to a product (CAUTION: ambiguous, historically a "GNU/Linux distribution"), composed of multiple test modules in a folder structure that compose test suites, e.g. "opensuse" (test distribution, short for "os-autoinst-distri-opensuse")
• product: the main "system under test" (SUT), e.g. "openSUSE"
• job group: equivalent to product, used in context of the webUI
• version: one version of a product, don't confuse with builds, e.g. "Tumbleweed"
• flavor: a specific variant of a product to distinguish differing variants, e.g. "DVD"
• arch: an architecture variant of a product, e.g. "x86_64"
• machine: additional variant of machine, e.g. used for "64bit", "uefi", etc.
• scenario: A composition of <distri>-<version>-<flavor>-<arch>-<test_suite>@<machine>, e.g. "openSUSE-Tumbleweed-DVD-x86_64-gnome@64bit", nicknamed koala
• build: Different versions of a product as tested, can be considered a "sub-version" of version, e.g. "Build1234"; CAUTION: ambiguity: either with the prefix "Build" included or not)

Thoughts about categorizing test results, issues, states within openQA¶

by okurz

When reviewing test results it is important to distinguish between different causes of "failed tests"

Nomenclature¶

Test status categories¶

A common definition about the status of a test regarding the product it tests: "false|true positive|negative" as described on https://en.wikipedia.org/wiki/False_positives_and_false_negatives. "positive|negative" describes the outcome of a test ("positive": PASSED; "negative": FAILED) whereas "false|true" describes the conclusion of the test regarding the presence of issues in the SUT or product in our case ("true": correct reporting; "false": incorrect reporting), e.g. "true negative", test successful, no issues detected and there are no issues, product is working as expected by customer. Another example: Think of testing as of a fire alarm. An alarm (event detector) should only go off (be "positive") if there is a fire (event to detect) --> "true positive" whereas if there is no fire there should be no alarm --> "true negative".

Another common but potentially ambiguous categorization:

• broken: the test is not behaving as expected (Ambiguity: "as expected" by whom?) --> commonly a "false positive", can also be "false negative" but hard to detect
• failing: the test is behaving as expected, but the test output is a fail --> "true positive"
• working: the test is behaving as expected (with no comment regarding the result, though some might ambiguously imply 'result is negative')
• passing: the test is behaving as expected, but the result is a success --> "true negative"

If in doubt declare a test as "broken". We should review the test and examine if it is behaving as expected.

Be careful about "positive/negative" as some might also use "positive" to incorrectly denote a passing test (and "negative" for failing test) as an indicator of "working product" not an indicator about "issue present". If you argue what is "used in common speech" think about how "false positive" is used as in "false alarm" --> "positive" == "alarm raised", also see https://narainko.wordpress.com/2012/08/26/understanding-false-positive-and-false-negative/

Priorization of work regarding categories¶

In this sense development+QA want to accomplish a "true negative" state whenever possible (no issues present, therefore none detected). As QA and test developers we want to prevent "false positives" ("false alarms" declaring a product as broken when it is not but the test failed for other reasons), also known as "type I error" and "false negatives" (a product issue is not catched by tests and might "slip through" QA and at worst is only found by an external outside customer) also known as "type II error". Also see https://en.wikipedia.org/wiki/Type_I_and_type_II_errors. In the context of openQA and system testing paired with screen matching a "false positive" is much more likely as the tests are very susceptible to subtle variations and changes even if they should be accepted. So when in doubt, create an issue in progress, look at it again, and find that it was a false alarm, rather than wasting more peoples time with INVALID bug reports by believing the product to be broken when it isn't. To quote Richard Brown: "I […] believe this is the route to ongoing improvement - if we have tests which produce such false alarms, then that is a clear indicator that the test needs to be reworked to be less ambiguous, and that IS our job as openQA developers to deal with".

Further categorization of statuses, issues and such in testing, especially automatic tests¶

By okurz

This categorization scheme is meant to help in communication in either written or spoken discussions being simple, concise, easy to remember while unambiguous in every case.
While used for naming it should also be used as a decision tree and can be followed from the top following each branch.

Categorization scheme¶

To keep it simple I will try to go in steps of deciding if a potential issue is of one of two categories in every step (maybe three) and go further down from there. The degree of further detailing is not limited, i.e. it can be further extended. Naming scheme should follow arabic number (for two levels just 1 and 2) counting schemes added from the right for every additional level of decision step and detail without any separation between the digits, e.g. "1111" for the first type in every level of detail up to level four. Also, I am thinking of giving the fully written form phonetic name to unambiguously identify each on every level as long as not more individual levels are necessary. The alphabet should be reserved for higher levels and higher priority types.
Every leaf of the tree must have an action assigned to it.

1 failed (ZULU)
11 new (passed->failed) (YANKEE)
111 product issue ("true positive") (WHISKEY)
1111 unfiled issue (SIERRA)
11111 hard issue (openqa fail) (KILO)
111121 critical / potential ship stopper (INDIA) --> immediately file bug report with "ship_stopper?" flag; opt. inform RM directly
111122 non-critical hard issue (HOTEL) --> file bug report
11112 soft issue (openqa softfail on job level, not on module level) (JULIETT) --> file bug report on failing test module
1112 bugzilla bug exists (ROMEO)
11121 bug was known to openqa / openqa developer --> cross-reference (bug->test, test->bug) AND raise review process issue, improve openqa process
11122 bug was filed by other sources (e.g. beta-tester) --> cross-reference (bug->test, test->bug)
112 test issue ("false positive") (VICTOR)
1121 progress issue exists (QUEBEC) --> cross-reference (issue->test, test->issue)
1122 unfiled test issue (PAPA)
11221 easy to do w/o progress issue
112211 need needles update --> re-needle if sure, TODO how to notify?
112212 pot. flaky, timeout
1122121 retrigger yields PASS --> comment in progress about flaky issue fixed
1122122 reproducible on retrigger --> file progress issue
11222 needs progress issue filed --> file progress issue
12 existing / still failing (failed->failed) (XRAY)
121 product issue (UNIFORM)
1211 unfiled issue (OSCAR) --> file bug report AND raise review process issue (why has it not been found and filed?)
1212 bugzilla bug exists (NOVEMBER) --> ensure cross-reference, also see rules for 1112 ROMEO
122 test issue (TANGO)
1221 progress issue exists (MIKE) --> monitor, if persisting reprioritize test development work
1222 needs progress issue filed (LIMA) --> file progress issue AND raise review process issue, see 1211 OSCAR
2 passed (ALFA)
21 stable (passed->passed) (BRAVO)
211 existing "true negative" (DELTA) --> monitor, maybe can be made stricter
212 existing "false negative" (ECHO) --> needs test improvement
22 fixed (failed->passed) (CHARLIE)
222 fixed "true negative" (FOXTROTT) --> TODO split monitor, see 211 DELTA
2221 was test issue --> close progress issue
2222 was product issue
22221 no bug report exists --> raise review process issue (why was it not filed?)
22222 bug report exists
222221 was marked as RESOLVED FIXED
221 fixed but "false negative" (GOLF) --> potentially revert test fix, also see 212 ECHO

Priority from high to low: INDIA->OSCAR->HOTEL->JULIETT->…

Further decision steps working on test issues¶

Test issues could be one of the following sources

• "accepted product changes"

  • product changed slightly but in an acceptable way without the need for communication with DEV+RM --> adapt test
  • product changed slightly but in an acceptable way found after feedback from RM --> adapt test
  • product changed significantly --> after approval by RM adapt test

• changes in test setup, e.g. our test hardware equipment behaves different or the network

• changes in test infrastructure software, e.g. os-autoinst, openQA

• changes in test management configuration, e.g. openQA database settings

• changes in the test software itself (the test plan in source code as well as needles)

• if none of the above it could be a hidden sporadic issue, i.e. the root problem is already hidden in the system for a long time but does not show symptoms every time

Advanced features in openQA¶

There are some features in openQA for reviewing test results and common practices. Some of these features are presented here based on the pull requests from github.

Show previous results in test results page gh#538¶

On a tests result page there is a tab for "previous results" showing the result of test runs in the same scenario. This shows previous builds as well as test runs in the same build. This way you can easily check and compare results from before including any comments, labels, bug references (see next section). This helps to answer questions like "Is this a new issue", "Is it reproducable", "has it been seen in before", "how does the history look like".

Querying the database for former test runs of the same scenario is a rather
costly operation which we do not want to do for multiple test results at once
but only for each individual test result (1:1 relation). This is why this is done in each individual test result and not for a complete build.

The evaluation of previous jobs is limited but can be adjusted with the query parameter limit_previous=<nr> in the test URL, e.g. to provide a link to the tab in the results page showing the previous 30 results of test 1234 on openqa.opensuse.org you would write
http://openqa.opensuse.org/tests/1234?limit_previous=30#previous

Remember that the higher the limit, the more complex the database queries will be increasing the lookup time as well as the load on openQA to generate the result.

Related issue: #10212

Screenshot of feature:

Link to latest in scenario name gh#836¶

Find the always latest job in a scenario with the link after the scenario name in the tab "Previous results"
Screenshot:

Add 'latest' query route gh#815¶

Should always refer to most recent job for the specified scenario.

• have the same link for test development, i.e. if one retriggers tests, the
  person has to always update the URL. If there would be a static URL even the
  browser can be instructed to reload the page automatically

• for linking to the always current execution of the last job within one
  scenario, e.g. to respond faster to the standard question in bug reports "does
  this bug still happen?"

Examples:

• tests/latest?distri=opensuse&version=13.1&flavor=DVD&arch=x86_64&test=kde&machine=64bit
• tests/latest?flavor=DVD&arch=x86_64&test=kde
• tests/latest?test=foobar - this searches for the most recent job using test_suite 'foobar' covering all distri, version, flavor, arch, machines. To be more specific, add the other query entries.

Add web UI controls to select 20/50/100/400 previous results gh#744¶

The query parameter 'limit_previous' allows to show more than the default 10
previous results on demand for some time. There are web UI
selections below the table of the previous build to reload the same page with
higher number of previous results on demand.

Example screenshot:

Show bug or label icon on overview if labeled gh#550¶

• Show bug icon with URL if mentioned in test comments
• Show bug or label icon on overview if labeled

For bugreferences write <bugtracker_shortname>#<bug_nr> in a comment, e.g. "bsc#1234", for generic labels use label:<keyword> where <keyword> can be any valid character up to the next whitespace, e.g. "false_positive". The keywords are not defined within openQA itself. A valid list of keywords should be decided upon within each project or environment of one openQA instance.

Example for a generic label:

Example for bug label:

Related issue: #10212

Hint: You can also write (or copy-paste) full links to bugs and issues. The links are automatically changed to the shortlinks (e.g. https://progress.opensuse.org/issues/11110 turns into poo#11110). Related issue: #11110

Also github pull requests and issues can be linked using the generic format
<marker>[#<project/repo>]#<id>, e.g. gh#os-autoinst/openQA#1234, see gh#973

Show certificate next to builds on overview if all failures are labeled gh#560, gh#1052¶

See online documentation about review badges.

Allow group overview query by result gh#531¶

This allows e.g. to show only failed builds. Could be included like in http://lists.opensuse.org/opensuse-factory/2016-02/msg00018.html for "known defects".

Example: Add query parameters like …&result=failed&arch=x86_64 to show only failed for the single architecture selected.

Add web UI controls to select more builds in group_overview gh#804¶

The query parameter 'limit_builds' allows to show more than the default 10
builds on demand. Just like we have for configuring previous results, the
current commit adds web UI selections to reload the same page with
higher number of builds on demand. For this, the limit of days is increased
to show more builds but still limited by the selected number.

Example screenshot:

Add more query parameters for configuring last builds gh#575¶

By using advanced query parameters in the URLs you can configure the search for builds.
Higher numbers would yield more complex database queries but can be selected
for special investigation use cases with the advanced query parameters, e.g. if one wants to get an overview of a longer history.
This applies to both the index dashboard and group overview page.

Example to show up to three week old builds instead of the default two weeks
with up to 20 builds instead of up to 10 being the default for the group
overview page:

http://openqa/group_overview/1?time_limit_days=21&limit_builds=20

Build tagging and keeping important builds gh#591¶

Tag builds with special comments on group overview¶

Based on comments on the group overview individual builds can be tagged. As
'build' by themselves do not own any data the job group is used to store this
information. A tag has a build to link it to a build. It also has a type
and an optional description. The type can later on be used to distinguish
tag types.

The generic format for tags is
tag:<build_id>:<type>[:<description>], e.g. tag:1234:important:Beta1.

The more recent tag always wins.

A 'tag' icon is shown next to tagged builds together with the description on
the group_overview page. The index page is not changed to prevent a potential
performance regression.

Within the sub group_overview the comments are parsed for comments and then
passed to the template explicitly to prevent duplicate database queries.

Keeping important builds¶

As builds can now be tagged we come up with the convention that the
'important' type - the only one for now - is used to tag every job that
corresponds to a build as 'important' and keep the logs for these jobs so that
we can always refer to the attached data, e.g. for milestone builds, final
releases, jobs for which long-lasting bug reports exist, etc.

As these jobs are not cleaned up automatically a manual or external cleanup
scheme has to be applied for important builds and jobs.

Example screenshot of a tag coment and corresponding tagged build¶

Related issue: #9544

Add web UI controls to filter only tagged or all builds gh#807¶

Using a new query parameter 'only_tagged=[0|1]' the list can be filtered, e.g. show only tagged (important) builds.

Example screenshot:

Related issue: #11052

Carry over bugrefs from previous jobs in same scenario if still failing gh#564¶

It is possible to label all failing tests but tedious to do by a human user
as many failures are just having the same issue until it gets fixed.
It helps if a label is preserved for a build that is still failing. This
idea is inspired by
https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin
and has been activated for bugrefs.

Does not carry over bugrefs over passes: After a job passed a new issue in a subsequent fail is assumed to be failed
for a different reason.

Related issue: #10212

Distinguish product and test issues bugref gh#708¶

"progress" is used to track test issues, bugzilla for product issues, at least for SUSE/openSUSE. openQA bugrefs distinguish this and show corresponding icons

Pinning comments as group description¶

This is possible by adding the keyword pinned-description anywhere in a comment on the group overview page. Then the comment will be shown at the top of the group overview page. However, it only works as operator or admin.

Filtering test results in test result overview¶

At the top of the test results overview page is a form which allows filtering tests by result, architecture and TODO-status.

Proposals for uses of labels¶

With Show bug or label icon on overview if labeled (gh#550) it is possible to add custom labels just by writing them. Nevertheless, a convention should be found for a common benefit. Beware that labels are also automatically carried over with (Carry over labels from previous jobs in same scenario if still failing [gh#564])(https://github.com/os-autoinst/openQA/pull/564) which might make consistent test failures less visible when reviewers only look for test results without labels or bugrefs. Labels are not anymore automatically carried over (gh#1071).

List of proposed labels with their meaning and where they could be applied.

• fixed_<build_ref>: If a test failure is already fixed in a more recent build and no bug reference is known, use this label together with a reference to a more recent passed test run in the same scenario. Useful for reviewing older builds. Example (https://openqa.suse.de/tests/382518#comments):

label:fixed_Build1501

t#382919

• needles_added: In case needles were missing for test changes or expected product changes caused needle matching to fail, use this label with a reference to the test PR or a proper reasoning why the needles were missing and how you added them. Example (https://openqa.suse.de/tests/388521#comments):

label:needles_added

needles for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/1353 were missing, added by jpupava in the meantime.