Introduction¶

This is the organisation wiki for the openQA Project.

Organisational¶

openQA calls¶

Currently there are two recurring openQA calls conducted at SUSE on http://jangouts.suse.de/. If there would be more interest from the outside the call could be made on a public platform.

Both meetings should target to finish in 15 minutes each. If more time is needed, propose to stay in the call with the required subset of attendees.

Standard rules of good "standup meetings" apply here, too, e.g.

• Be on time (be there at meeting start)
• Be concise (help keep the time limit)
• Be polite
• focus on
  • what you achieved
  • what you plan
  • where did you face problems where you could use help

"openQA backend coordination" call¶

objectives:

• Coordination on openQA backend development

execution: A regular daily call from Mon-Fri at 0900 UTC

"SUSE QA test coordination" call¶

objectives:

• Coordination on openQA based test development, especially SLE products
• Information about important development in openQA backend by backend responsibles

execution: Mon + Wed, at 0930 UTC

If somebody from SUSE QA team will do back-end development he can attend the first call as well, of course.

User stories¶

User story 1¶

User: QA-Project Managment
primary actor: QA Project Manager, QA Team Leads
stakeholder: Directors, VP
trigger: product milestones, providing a daily status
user story: „As a QA project manager I want to check on a daily basis the „openQA Dashboard“ to get a summary/an overall status of the „reviewers results“ in order to take the right actions and prioritize tasks in QA accordingly.“

User story 2¶

User: openQA-Admin
primary actor: Backend-Team
stakeholder: Qa-Prjmgr, QA-TL, openQA Tech-Lead
trigger: Bugs, features, new testcases
user story: „As an openQA admin I constantly check in the web-UI the system health and I manage its configuration to ensure smooth operation of the tool.“

User story 3¶

User: QA-Reviewer
primary actor: QA-Team
stakeholder: QA-Prjmgr, Release-Mgmt, openQA-Admin
trigger: every new build
user story: „As an openQA-Reviewer at any point in time I review on the webpage of openQA the overall status of a build in order to track and find bugs, because I want to find bugs as early as possible and report them.“

User story 4¶

User: Testcase-Contributor
primary actor: All development teams, Maintenance QA
stakeholder: QA-Reviewer, openQA-Admin, openQA Tech-Lead
trigger: features, new functionality, bugs, new product/package
user story: 4. „As developer when there are new features, new functionality, bugs, new product/package in git I contribute my testcases because I want to ensure good quality submissions and smooth product integration.“

User story 5¶

User: Release-Mgmt
primary actor: Release Manager
stakeholder: Directors, VP, PM, TAMs, Partners
trigger: Milestones
user story: „As a Release-Manager on a daily basis I check on a dashboard for the product health/build status in order to act early in case of failures and have concrete and current reports.“

User story 6¶

User: Staging-Admin
primary actor: Staging-Manager for the products
stakeholder: Release-Mgmt, Build-Team
trigger: every single submission to projects
user story: „As a Staging-Manager I review the build status of packages with every staged submission to the „staging projects“ in the „staging dashboard“ and the test-status of the pre-integrated fixes, because I want to identify major breakage before integration to the products and provide fast feedback back to the development.“

Thoughts about categorizing test results, issues, states within openQA¶

by okurz

When reviewing test results it is important to distinguish between different causes of "failed tests"

Nomenclature¶

Test status categories¶

A common definition about the status of a test regarding the product it tests:

• broken: the test is not behaving as expected (Ambiguity: "as expected" by whom?)
• failing: the test is behaving as expected, but the result is negative
• working: the test is behaving as expected (with no comment regarding the result, though some might ambiguously imply 'result is positive')
• passing: the test is behaving as expected, but the result is positive

If in doubt declare a test as "broken". We should review the test and examine if it is behaving as expected.

Further nomenclature is "false|true positive|negative" as described on https://en.wikipedia.org/wiki/False_positives_and_false_negatives. "false|true" describes the status of the SUT or product in our case ("true": product is working as expected by customer, "false" otherwise), "positive|negative" describes the status of the test ("positive": test passed, "negative" otherwise).

Priorization of work regarding categories¶

In this sense we want to accomplish a "true positive" state whenever possible. As QA and test developers we want to prevent "true negatives" ("false alarms" declaring a product as broken when it is not but the test failed for other reasons) and "false positives" (a product issue is not catched by tests and might "slip through" QA and at worst is only found by an external outside customer). In the context of openQA and system testing paired with screen matching a "true negative" is much more likely. So when in doubt, create an issue in progress, look at it again, and find that it was a false alarm, rather than wasting more peoples times with INVALID bug reports by believing the product to be broken when it isn't. To quote Richard Brown: "I […] believe this is the route to ongoing improvement - if we have tests which produce such false alarms, then that is a clear indicator that the test needs to be reworked to be less ambiguous, and that IS our job as openQA developers to deal with"

Old content¶

Sprints¶

Sprint 01
Sprint 02
Sprint 03