Introduction¶

This is the organisation wiki for the openQA Project.

Organisational¶

openQA calls¶

Currently there are two recurring openQA calls conducted at SUSE on http://jangouts.suse.de/. If there would be more interest from the outside the call could be made on a public platform.

Both meetings should target to finish in 15 minutes each. If more time is needed, propose to stay in the call with the required subset of attendees.

Standard rules of good "standup meetings" apply here, too, e.g.

• Be on time (be there at meeting start)
• Be concise (help keep the time limit)
• Be polite
• focus on
  • what you achieved
  • what you plan
  • where did you face problems where you could use help

"openQA backend coordination" call¶

objectives:

• Coordination on openQA backend development

execution: A regular daily call from Mon-Fri at 0900 UTC

"SUSE QA test coordination" call¶

objectives:

• Coordination on openQA based test development, especially SLE products
• Information about important development in openQA backend by backend responsibles

execution: Mon + Wed, at 0930 UTC

If somebody from SUSE QA team will do back-end development he can attend the first call as well, of course.

User stories¶

User story 1¶

User: QA-Project Managment
primary actor: QA Project Manager, QA Team Leads
stakeholder: Directors, VP
trigger: product milestones, providing a daily status
user story: „As a QA project manager I want to check on a daily basis the „openQA Dashboard“ to get a summary/an overall status of the „reviewers results“ in order to take the right actions and prioritize tasks in QA accordingly.“

User story 2¶

User: openQA-Admin
primary actor: Backend-Team
stakeholder: Qa-Prjmgr, QA-TL, openQA Tech-Lead
trigger: Bugs, features, new testcases
user story: „As an openQA admin I constantly check in the web-UI the system health and I manage its configuration to ensure smooth operation of the tool.“

User story 3¶

User: QA-Reviewer
primary actor: QA-Team
stakeholder: QA-Prjmgr, Release-Mgmt, openQA-Admin
trigger: every new build
user story: „As an openQA-Reviewer at any point in time I review on the webpage of openQA the overall status of a build in order to track and find bugs, because I want to find bugs as early as possible and report them.“

User story 4¶

User: Testcase-Contributor
primary actor: All development teams, Maintenance QA
stakeholder: QA-Reviewer, openQA-Admin, openQA Tech-Lead
trigger: features, new functionality, bugs, new product/package
user story: 4. „As developer when there are new features, new functionality, bugs, new product/package in git I contribute my testcases because I want to ensure good quality submissions and smooth product integration.“

User story 5¶

User: Release-Mgmt
primary actor: Release Manager
stakeholder: Directors, VP, PM, TAMs, Partners
trigger: Milestones
user story: „As a Release-Manager on a daily basis I check on a dashboard for the product health/build status in order to act early in case of failures and have concrete and current reports.“

User story 6¶

User: Staging-Admin
primary actor: Staging-Manager for the products
stakeholder: Release-Mgmt, Build-Team
trigger: every single submission to projects
user story: „As a Staging-Manager I review the build status of packages with every staged submission to the „staging projects“ in the „staging dashboard“ and the test-status of the pre-integrated fixes, because I want to identify major breakage before integration to the products and provide fast feedback back to the development.“

Thoughts about categorizing test results, issues, states within openQA¶

by okurz

When reviewing test results it is important to distinguish between different causes of "failed tests"

Nomenclature¶

Test status categories¶

A common definition about the status of a test regarding the product it tests: "false|true positive|negative" as described on https://en.wikipedia.org/wiki/False_positives_and_false_negatives. "positive|negative" describes the outcome of a test ("positive": PASSED; "negative": FAILED) whereas "false|true" describes the conclusion of the test regarding the presence of issues in the SUT or product in our case ("true": correct reporting; "false": incorrect reporting), e.g. "true negative", test successful, no issues detected and there are no issues, product is working as expected by customer. Another example: Think of testing as of a fire alarm. An alarm (event detector) should only go off (be "positive") if there is a fire (event to detect) --> "true positive" whereas if there is no fire there should be no alarm --> "true negative".

Another common but potentially ambiguous categorization:

• broken: the test is not behaving as expected (Ambiguity: "as expected" by whom?) --> commonly a "false positive", can also be "false negative" but hard to detect
• failing: the test is behaving as expected, but the test output is a fail --> "true positive"
• working: the test is behaving as expected (with no comment regarding the result, though some might ambiguously imply 'result is negative')
• passing: the test is behaving as expected, but the result is a success --> "true negative"

If in doubt declare a test as "broken". We should review the test and examine if it is behaving as expected.

Be careful about "positive/negative" as some might also use "positive" to incorrectly denote a passing test (and "negative" for failing test) as an indicator of "working product" not an indicator about "issue present". If you argue what is "used in common speech" think about how "false positive" is used as in "false alarm" --> "positive" == "alarm raised", also see https://narainko.wordpress.com/2012/08/26/understanding-false-positive-and-false-negative/

Priorization of work regarding categories¶

In this sense development+QA want to accomplish a "true negative" state whenever possible (no issues present, therefore none detected). As QA and test developers we want to prevent "false positives" ("false alarms" declaring a product as broken when it is not but the test failed for other reasons), also known as "type I error" and "false negatives" (a product issue is not catched by tests and might "slip through" QA and at worst is only found by an external outside customer) also known as "type II error". Also see https://en.wikipedia.org/wiki/Type_I_and_type_II_errors. In the context of openQA and system testing paired with screen matching a "false positive" is much more likely as the tests are very susceptible to subtle variations and changes even if they should be accepted. So when in doubt, create an issue in progress, look at it again, and find that it was a false alarm, rather than wasting more peoples time with INVALID bug reports by believing the product to be broken when it isn't. To quote Richard Brown: "I […] believe this is the route to ongoing improvement - if we have tests which produce such false alarms, then that is a clear indicator that the test needs to be reworked to be less ambiguous, and that IS our job as openQA developers to deal with".

Old content¶

Sprints¶

Sprint 01
Sprint 02
Sprint 03