Wiki » History » Revision 417
Revision 416 (okurz, 2022-01-25 09:30) → Revision 417/424 (okurz, 2022-01-25 10:33)
{{toc}}
# Test results overview
* Latest report based on openQA test results for [OSD](https://openqa.io.suse.de/openqa-review/openqa_suse_de_status.html) and [SLE15](https://openqa.io.suse.de/openqa-review/openqa_sle15_status.html)
* Find more reports on https://openqa.io.suse.de/openqa-review/
* only "blocker" or "shipstopper" bugs on "interesting products" for SLE: http://s.qa.suse.de/qa_sle_bugs_sle , SLE15: http://s.qa.suse.de/qa_sle_bugs_sle15_all, SLE12: http://s.qa/qa_sle_bugs_sle12_2
# QE tools - Team description
"The easiest way to provide complete quality for your software"
We provide the most complete free-software system-level testing solution to ensure high quality of operating systems, complete software stacks and multi-machine services for software distribution builders, system integration engineers and release teams. We continuously develop, maintain and release our software to be readily used by anyone while we offer a friendly community to support you in your needs. We maintain the main public and SUSE internal openQA server as well as supporting tools in the surrounding ecosystem.
## Team responsibilities
* Develop and maintain upstream openQA including the backend os-autoinst
* Administration of openqa.suse.de and workers (But not physical hardware, as these belong to the departments that purchased them and we merely facilitate)
* Helps administrating and maintaining openqa.opensuse.org, including coordination of efforts aiming at solving problems affecting o3
* Develop and maintain SUSE maintenance QA tools (SMELT, template generator, MTUI, openQA QAM bot, etc, e.g. from https://confluence.suse.com/display/maintenanceqa/Toolchain+for+maintenance+quality+engineering)
* Help with the investigation of specific issues, especially when they are likely related to generic or backend problems
* Support colleagues, team members and open source community
## Out of scope
* Maintenance and recurring review of individual tests
* Maintenance of physical hardware
* Maintenance of special worker addendums needed for tests, e.g. external hypervisor hosts for s390x, powerVM, xen, hyperv, IPMI, VMWare (Clarification: We maintain the code for all backends but we are no experts in specific domains. So we always try to help but it's a case by case decision based on what we realistically can provide based on our competence.)
* Ticket triaging of http://progress.opensuse.org/projects/openqatests/
* Setup of configuration for individual products to test, e.g. new job groups in openQA
* Feature development within the backend for single teams (commonly provided by teams themselves)
## Our common userbase
Known users of our products: Most SUSE QA engineers, SUSE SLE release managers and release engineers, every SLE developer submitting "submit requests" in OBS/IBS where product changes are tested as part of the "staging" process before changes are accepted in either SLE or openSUSE (staging tests must be green before packages are accepted), same for all openSUSE contributors submitting to either openSUSE:Factory (for Tumbleweed, SLE, future Leap versions) or Leap, other GNU/Linux distributions like Fedora https://openqa.fedoraproject.org/ , Debian https://openqa.debian.net/ , https://openqa.qubes-os.org/ , https://openqa.endlessm.com/ , the GNOME project https://openqa.gnome.org, https://www.codethink.co.uk/articles/2021/automated-linux-kernel-testing/, openSUSE KDE contributors (with their own workflows, https://openqa.opensuse.org/group_overview/23 ), openSUSE GNOME contributors (https://openqa.opensuse.org/group_overview/35 ), OBS developers (https://openqa.opensuse.org/parent_group_overview/7#grouped_by_build) , wicked developers (https://gitlab.suse.de/wicked-maintainers/wicked-ci#openqa), and of course our team itself for "openQA-in-openQA Tests" :) https://openqa.opensuse.org/group_overview/24
Keep in mind: "Users of openQA" and talking about "openSUSE release managers and engineers" means SUSE employees but also employees of other companies, also development partners of SUSE.
In summary our products, for example openQA, are a critical part of many development processes hence outages and regressions are disruptive and costly. Hence we need to ensure a high quality in production hence we practice DevOps with a slight tendency to a conservative approach for introducing changes while still ensuring a high development velocity.
## How we work
The QE Tools team is following the DevOps approach working using a lightweight Agile approach also inspired by [Extreme Programming](https://extremeprogramming.org/) and [Kanban](https://en.wikipedia.org/wiki/Kanban_(development)) and of course the original http://agilemanifesto.org/. We plan and track our works using tickets on https://progress.opensuse.org . We pick tickets based on priority and planning decisions. We use weekly meetings as checkpoints for progress and also track cycle and lead times to crosscheck progress against expectations.
* [tools team - backlog](https://progress.opensuse.org/issues?query_id=230): The complete backlog of the team
* [tools team - backlog, high-level view](https://progress.opensuse.org/issues?query_id=526): A high-level view of the backlog, all epics and higher (an "epic" includes multiple stories)
* [tools team - backlog, top-level view](https://progress.opensuse.org/issues?query_id=524): A top-level view of the backlog, only sagas and higher (a "saga" is bigger than an epic and can include multiple epics, i.e. "epic of epics")
* [tools team - what members of the team are working on](https://progress.opensuse.org/issues?query_id=400): To check progress and know what the team is currently occupied with
* [tools team - closed within last 60 days](https://progress.opensuse.org/issues?query_id=541): What was recently resolved
*Be aware:* Custom queries in the right-hand sidebar of individual projects, e.g. https://progress.opensuse.org/projects/openqav3/issues , show queries with the same name but are limited to the scope of the specific projects so can show only a subset of all relevant tickets.
### What we expect from team members
* Actively show visible contributions to our products every workday *(pull requests, code review, ticket updates in decending priority, i.e. if you are very active in pull requests + code review ticket updates are much less important)*
* Be responsive over usual communication platforms and channels *(user questions, team discussions)*
* Stick to our rules *(this wiki, SLOs, alert handling)*
### Common tasks for team members
This is a list of common tasks that we follow, e.g. reviewing daily based on individual steps in the DevOps Process 
* **Plan**:
* State daily learning and planned tasks in internal chat room
* Review backlog for time-critical, triage new tickets, pick tickets from backlog; see https://progress.opensuse.org/projects/qa/wiki#How-we-work-on-our-backlog
* **Code**:
* See project specific contribution instructions
* Provide peer-review following https://github.com/notifications based on projects within the scope of https://github.com/os-autoinst with the exception of test code repositories, especially https://github.com/os-autoinst/openQA, https://github.com/os-autoinst/os-autoinst, https://github.com/os-autoinst/scripts, https://github.com/os-autoinst/os-autoinst-distri-openQA, https://github.com/os-autoinst/openqa-trigger-from-obs, https://github.com/os-autoinst/openqa_review as well as other projects like https://github.com/openSUSE/Mojo-IOLoop-ReadWriteProcess and https://gitlab.suse.de/qa-maintenance/openQABot
* **Build**:
* See project specific contribution instructions
* **Test**:
* Monitor failures on https://travis-ci.org/ relying on https://build.opensuse.org/package/show/devel:openQA/os-autoinst_dev for os-autoinst (email notifications)
* Monitor failures on https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master relying on https://build.opensuse.org/project/show/devel:openQA:ci for openQA (email notifications)
* **Release**:
* By default we use the rolling-release model for all projects unless specified otherwise
* Monitor [devel:openQA on OBS](https://build.opensuse.org/project/show/devel:openQA) (all packages and all subprojects) for failures, ensure packages are published on http://download.opensuse.org/repositories/devel:/openQA/, ensure to be added as a Maintainer for that project (members need to be added individually, you can ask existing team members, e.g. the SM)
* Monitor http://jenkins.qa.suse.de/view/openQA-in-openQA/ for the openQA-in-openQA Tests and automatic submissions of os-autoinst and openQA to openSUSE:Factory through https://build.opensuse.org/project/show/devel:openQA:tested
* **Deploy**:
* o3 is automatically deployed (daily), see https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Automatic-update-of-o3
* osd is automatically deployed (multiple times per week), monitor https://gitlab.suse.de/openqa/osd-deployment/pipelines and watch for notification email to openqa@suse.de
* **Operate**:
* Apply infrastructure changes from https://gitlab.suse.de/openqa/salt-states-openqa (osd) or manually over sshd (o3)
* Monitor for backup, see https://gitlab.suse.de/qa-sle/backup-server-salt
config changes in salt (osd), backups, job group configuration changes
* Ensure old unused/non-matching needles are cleaned up (osd+o3), see #73387
* Maintain https://gitlab.suse.de/qa-maintenance/qamops and https://confluence.suse.com/display/maintenanceqa/qam.suse.de
* **Monitor**:
* React on alerts from [stats.openqa-monitor.qa.suse.de](https://stats.openqa-monitor.qa.suse.de/alerting/list?state=not_ok) (emails on [osd-admins@suse.de](http://mailman.suse.de/mailman/listinfo/osd-admins) and login via LDAP credentials, you must be an *editor* to edit panels and hooks via the web UI)
* Look for incomplete jobs or scheduled not being worked on o3 and osd (API or webUI) - see also #81058 for *power*
* React on alerts from https://gitlab.suse.de/openqa/auto-review/, https://gitlab.suse.de/openqa/openqa-review/, https://gitlab.suse.de/openqa/monitor-o3 (subscribe to projects for notifications)
* Be responsive on #opensuse-factory (irc://irc.libera.chat/opensuse-factory, formerly irc://chat.freenode.net/opensuse-factory) for help, support and collaboration (Unless you have a better solution it is suggested to use [Element.io](https://matrix.to/#/!ilXMcHXPOjTZeauZcg:libera.chat) for a sustainable presence; you also need a [registered IRC account](https://libera.chat/guides/registration), formerly [freenode](https://freenode.net/kb/answer/registration))
* Be responsive on [#qa-tools in Rocket.Chat](https://chat.suse.de/channel/qa-tools) for internal coordination and alarm handling, fallback to #suse-qe-tools:opensuse.org (matrix) as backup if other channels are temporarily down, alternatively public channels on matrix/ IRC if the topics are not confidential
* Be responsive on [#testing](https://chat.suse.de/channel/testing) for help, support and collaboration
* Be responsive on mailing lists opensuse-factory@opensuse.org and openqa@suse.de (see https://en.opensuse.org/openSUSE:Mailing_lists_subscription)
* Be responsive in https://matrix.to/#/#openqa:opensuse.org or the bridged room [#openqa](https://discord.com/channels/366985425371398146/817367056956653621) on https://discord.gg/opensuse if you have a discord account
### How we work on our backlog
* "due dates" are only used as exception or reminders
* every team member can pick up tickets themselves
* everybody can set priority, PO can help to resolve conflicts
* consider the [ready, not assigned/blocked/low](https://progress.opensuse.org/issues?query_id=490) query as preferred. It is suggested to pick up tickets based on priority. "Workable" tickets are often convenient and hence preferred.
* ask questions in tickets, even potentially "stupid" questions, oftentimes descriptions are unclear and should be improved
* There are "low-level infrastructure tasks" only conducted by some team members, the "DevOps" aspect does not include that but focusses on the joint development and operation of our main products
* Consider tickets with the subject keyword or tag "learning" as good learning opportunities for people new to a certain area. Experts in the specific area should prefer helping others but not work on the ticket
* For tickets which are out of the scope of the team remove from backlog, delegate to corresponding teams or persons but be nice and supportive, e.g. [SUSE-IT](https://sd.suse.com/), [EngInfra](https://infra.nue.suse.com/) also see [SLA](https://confluence.suse.com/display/qasle/Service+Level+Agreements), [test maintainer](https://progress.opensuse.org/projects/openqatests/), QE-LSG PrjMgr/mgmt
* For [EngInfra tickets](https://sd.suse.com/servicedesk/customer/portal/1) ensure there's a ticket for it in [openQA Infrastructure](https://progress.opensuse.org/projects/openqa-infrastructure/issues), use `EngInfra` under **Select a system**, use `[openqa] …` in the subject, reference the progress ticket. After the ticket was created and was auto-assigned to "Eng-Infa" (sic!) then **Share** with `OSD Admins` (the icon with two figures, not a single gray avatar) after creating the ticket (Sharing the ticket on creation time does not work because the group will be overwritten after "suse-admin" auto-assigns to a different queue). Use the tracker ticket for internal notes
* Whenever we apply changes to the infrastructure we should have a ticket
* Refactoring and general improvements are conducted while we work on features or regression fixes
* For every regression or bigger issue that we encounter try to come up with at least two improvements, e.g. the actual issue is fixed and similar cases are prevented in the future with better tests and optionally also monitoring is improved
* For critical issues and very big problems especially when we were informed by users about outages collect "lessons learned", e.g. in notes in the ticket or a meeting with minutes in the ticket, consider https://en.wikipedia.org/wiki/Five_whys and answer at least the following questions: "User impact, outwards-facing communication and mitigation, upstream improvement ideas, Why did the issue appear, can we reduce our detection time, can we prevent similar issues in the future, what can we improve technically, what can we improve in our processes". Also see https://youtu.be/_Dv4M39Arec
* okurz proposes to use "#NoEstimates". Though that topic is controversial and often misunderstood. https://ronjeffries.com/xprog/articles/the-noestimates-movement/ describes it nicely :) Hence tickets should be evenly sized and no estimation numbers should be provided on tickets
* If you really want you can look at the [burndown chart](https://progress.opensuse.org/agile/charts?utf8=%E2%9C%93&set_filter=1&f%5B%5D=chart_period&op%5Bchart_period%5D=%3E%3Ct-&v%5Bchart_period%5D%5B%5D=90&f%5B%5D=fixed_version_id&op%5Bfixed_version_id%5D=%3D&v%5Bfixed_version_id%5D%5B%5D=418&f%5B%5D=&chart=burndown_chart&chart_unit=issues&interval_size=day) (some people wish to have this) but we consider it unnecessary due to the continuous development, not a project with defined end. Also an [agile board](https://progress.opensuse.org/agile/board?utf8=%E2%9C%93&set_filter=1&f%5B%5D=fixed_version_id&op%5Bfixed_version_id%5D=%3D&v%5Bfixed_version_id%5D%5B%5D=418&f%5B%5D=status_id&op%5Bstatus_id%5D=%3D&f_status%5B%5D=1&f_status%5B%5D=12&f_status%5B%5D=2&f_status%5B%5D=15&f_status%5B%5D=4&c%5B%5D=tracker&c%5B%5D=assigned_to&c%5B%5D=cf_16) is available but likely due to problems within the redmine installation ordering cards is not reliable.
* Write to qa-team@suse.de as well for critical changes as well as chat channels
* Everyone should propose reverts of features if we find problems that can not be immediately fixed or worked around in production
#### Definition of DONE
Also see https://web.archive.org/web/20110308065330/http://www.allaboutagile.com/definition-of-done-10-point-checklist/ and https://web.archive.org/web/20170214020537/https://www.scrumalliance.org/community/articles/2008/september/what-is-definition-of-done-(dod)
* [[Tools]] Code changes are made available via a pull request on a version control repository, e.g. github for openQA
* [Guidelines for git commits](http://chris.beams.io/posts/git-commit/) have been followed
* Code has been reviewed (e.g. in the github PR)
* Depending on criticality/complexity/size/feature: A local verification test has been run, e.g. post link to a local openQA machine or screenshot or logfile
* For regressions: A regression fix is provided, flaws in the design, monitoring, process have been considered
* Potentially impacted package builds have been considered, e.g. openSUSE Tumbleweed and Leap, Fedora, etc.
* Code has been merged (either by reviewer or "mergify" bot or reviewee after 'LGTM' from others)
* Code has been deployed to osd and o3 (monitor automatic deployment, apply necessary config or infrastructure changes)
#### Definition of READY for new features
The following points should be considered before a new feature ticket is READY to be implemented:
* Follow the ticket template from https://progress.opensuse.org/projects/openqav3/wiki/#Feature-requests
* A clear motivation or user expressing a wish is available
* Acceptance criteria are stated (see ticket template) or use `[timeboxed:<nr>h]` with `<nr>` hours for tasks that should be limited in time, e.g. a research task with `[timeboxed:20h] research …`
* add tasks as a hint where to start
#### WIP-limits (reference "Kanban development")
* global limit of 10 tickets, and 3 tickets per person respectively [In Progress](https://progress.opensuse.org/issues?query_id=505)
* limit of 20 tickets per person in [Feedback](https://progress.opensuse.org/issues?query_id=520)
#### Target numbers or "guideline", "should be", in priorities
1. *New, untriaged QA (openQA, etc.):* [0 (daily)](https://progress.opensuse.org/projects/qa/issues?query_id=576) . Every ticket should have a target version, e.g. "Ready" for QE tools team, "future" if unplanned, others for other teams
1. *Untriaged "tools" tagged:* [0 (daily)](https://progress.opensuse.org/issues?query_id=481) . Every ticket should have a target version, e.g. "Ready" for QE tools team, "future" if unplanned, others for other teams
1. *Workable (properly defined):* [10-40](https://progress.opensuse.org/issues?query_id=478) . Enough tickets to reflect a proper plan but not too many to limit unfinished data (see "waste")
1. *Overall backlog length:* [ideally less than 100](https://progress.opensuse.org/issues?query_id=230) . Similar as for "Workable". Enough tickets to reflect a proper roadmap as well as give enough flexibility for all unfinished work but limited to a feasible number that can still be overlooked by the team without loosing overview. One more reason for a maximum of 100 are that pagination in redmine UI allows to show only up to 100 issues on one page at a time, same for redmine API access.
1. *Within due-date:* [0 (daily/weekly)](https://progress.opensuse.org/issues?query_id=514) . We should take due-dates serious, finish tickets fast and at the very least update tickets with an explanation why the due-date could not be hold and update to a reasonable time in the future based on usual cycle time expectations
#### SLOs (service level objectives)
* for picking up tickets based on priority, first goal is "urgency removal":
* **immediate**: [<1 day](https://progress.opensuse.org/issues?query_id=542)
* **urgent**: [<1 week](https://progress.opensuse.org/issues?query_id=543)
* **high**: [<1 month](https://progress.opensuse.org/issues?query_id=544)
* **normal**: [<1 year](https://progress.opensuse.org/issues?query_id=545)
* **low**: undefined
* aim for cycle time of individual tickets (not epics or sagas): 1h-2w
* reference for SLOs and related topics: https://sre.google/sre-book/table-of-contents/
#### Backlog prioritization
When we prioritize tickets we assess:
1. What the main use cases of openQA are among all users, be it SUSE QA engineers, other SUSE employees, openSUSE contributors as well as any other outside user of openQA
2. We try to understand how many persons and products are affected by feature requests as well as regressions (or "concrete bugs" as the ticket category is called within the openQA Project) and prioritize issues affecting more persons and products and use cases over limited issues
3. We prioritize regressions higher than work on (new) feature requests
4. If a workaround or alternative exists then this lowers priority. We prioritize tasks that need deep understanding of the architecture and an efficient low-level implementation over convenience additions that other contributors are more likely to be able to implement themselves.
#### Periodic backlog grooming
These queries can be used as help to organize our work efficiently
* [QE tools team - backlog - sorted by update time](https://progress.opensuse.org/issues?query_id=654) ensure all tickets are reasonably up-to-date and don't keep hanging around
* [QE tools team - due date forecast](https://progress.opensuse.org/issues?query_id=651) prevent running into due-dates proactively
### Team meetings
* **Daily:** Use (internal) chat actively, e.g. formulate your findings or achievements and plans for the day, "think out loud" while working on individual problems. Optionally join [m.o.o/suse_qa_tools](https://meet.opensuse.org/suse_qa_tools) every Monday, Tuesday and Thursday 1030-1045 CET/CEST
* *Goal*: Emergency responses, clarify next steps or blockers on current work items, asking and answering questions on tickets that would be ignored otherwise, ticket estimations (after the regular daily) (compare to [Daily Scrum](https://www.scrumguides.org/scrum-guide.html#events-daily))
* **Ticket estimations:** Every Thursday 1110-1210 CET/CEST in [m.o.o/suse_qa_tools](https://meet.opensuse.org/suse_qa_tools) Estimate t-shirt sizes for our tickets
* *Goal*: Ensure tickets are workable. Refine and split tickets for larger estimates.
* **Midweekly unblock:** Every Wednesday 1110-1210 CET/CEST in [m.o.o/suse_qa_tools](https://meet.opensuse.org/suse_qa_tools).
* *Goal*: Discuss tasks in progress in more detail, unblock people.
* **Weekly coordination:** Every Friday 1110-1140(-1210) CET/CEST in [m.o.o/suse_qa_tools](https://meet.opensuse.org/suse_qa_tools). Community members and guests are particularly welcome to join this meeting.
* *Goal*: Demo of features, Team backlog coordination and design decisions of bigger topics (compare to [Sprint Planning](https://www.scrumguides.org/scrum-guide.html#events-planning)).
* *Conduction*: Demo recently finished feature work depending on [last closed](https://progress.opensuse.org/issues?query_id=572), crosscheck status of team, discuss blocked tasks and upcoming work
* **Fortnightly Retrospective:** Friday 1140-1210 CET/CEST every odd week, same room as the weekly meeting. On these days the weekly has hard time limit of 1110-1140.
* *Goal*: Inspect and adapt, learn and improve (compare to [Sprint Retrospective](https://www.scrumguides.org/scrum-guide.html#events-retro))
* *Announcements*: Create a new *discussion* with all team members in Rocket Chat and a new [retrospected game](retrospected.com) which can be filled in all week. Specific actions will be recorded as tickets.
* **Virtual coffee:** Weekly every Monday 1100-1120 CET/CEST, same room as the weekly.
* *Goal*: Connect and bond as a team, understand each other (compare to [Informal Communication in an all-remote environment](https://about.gitlab.com/company/culture/all-remote/informal-communication))
* **extension on-demand:** Optional meeting on invitation in the suggested time slot Thursday 1000-1200 CET/CEST, in the same room as the weekly, on-demand or replacing the *Virtual coffee talk*.
* *Goal*: Introduce, research and discuss bigger topics, e.g. backlog overview, processes and workflows
* **Workshop:** Friday 0900-0950 CET/CEST every week in [m.o.o/suse_qa_tools](https://meet.opensuse.org/suse_qa_tools) especially for community members and users! We will run this every week with the plan to move to a fortnightly cadence every even week.
* *Goal*: Demonstrate new and important features, explain already existing, but less well-known features, and discuss questions from the user community. All your questions are welcome!
* *Announcements*: Drop a reminder with a teaser in [#testing](https://chat.suse.de/channel/testing).
* *Recordings*: Consider recording, e.g. using OBS, and upload to youtube, link on topics link. SUSE internal topics can be published on http://streaming.nue.suse.com/i/QE-Tools-Workshops/ by ssh-uploading to ftp@streaming.nue.suse.com:~/i/QE-Tools-Workshops/ (get your SSH key added by existing team members, e.g. okurz)
**NOTICE**: We're are currently using meet.opensuse.org (m.o.o). As fallback on problems use [fallback](https://meet.jit.si/suse_qa_tools)
#### Best practices for meetings
* Meetings concerning the whole team are moderated by the scrum master by default, who should join the call early and verify that the meeting itself and any tools used are working or e.g. advise the use of the fallback option.
* We would prefer UTC for meeting times to be globally fair but as many other SUSE meetings are bound to European time we need to stick to that as well.
* It is recommended to use the Jitsi Audio-feedback feature, blue/green circles depending on microphone volume. Everybody should ensure that at least "two green balls" show up
* Hand signals over video can be used, e.g. "waving/circling hands": "I am lost, please bring me into discussion again"; "T-Sign": "I need a break"; "Raised hand": "I would like to speak"
* Discuss topics relevant for all within the common meetings, continue discussions pro-actively over asynchronous communication, e.g. tickets, as well as conduct topic centered follow-up meetings with only relevant attendees
* Reminders in Slack correct for summer/winter time automatically but if you make changes on them the time might be shifted by one hour e.g. if you scheduled a reminder on 10:30 am CEST, it will become 9:30 CET after the switch
* Use https://etherpad.opensuse.org/p/suse_qe_tools for collaborative editing and put the content back into tickets or wikis
#### Workshop Topics
* *SUSE QE Tools roadmap*: Recent achievements, mid-term plan and future outlook. Every first Friday every month (Idea based on discussion between okurz and vpelcak 2021-02-09)
* **2021-01-15:** *DONE* [openqa-auto-review and openqa-investigate](https://youtu.be/_t3THhdiDag)
* **2021-01-29:** *DONE* overview of development repositories on https://github.com/os-autoinst/
* **2021-02-05:** *DONE* [powerpc](https://youtu.be/q1CM2AH5aKY) (@nicksinger)
* **2021-02-12:** *DONE* [job templates](https://youtu.be/YPuH0bcr524) (@tinita, @cdywan)
* **2021-02-19:** *DONE* [SUSE QEM review workflow discussions](https://youtu.be/nCIAcvD7SA8) (@dzedro, @mgrifalconi)
* **2021-02-26:** *DONE* open conversation
* **2021-03-05:** *DONE* [SUSE QE Tools roadmap - 2021-03](https://youtu.be/vIqBIEMH0O0) (@okurz, @mkittler)
* **2021-03-12:** *DONE* [openqa-mon](https://youtu.be/CNLihgMKt30) @ph03nix
* **2021-03-19:** *DONE* [multi-machine tests](https://youtu.be/9j-NgNTzJ0w) (@okurz; topic proposal by zluo, initially brought up as: "high RAM and storage requirements")
* **2021-03-26:** *skipped due to SUSE Hack Week*
* **2021-04-02:** *public holiday*
* **2021-04-09:** *DONE* [SUSE QE Tools roadmap - 2021-04](https://youtu.be/nfMilLcCosQ) (@okurz, @cdywan)
* **2021-04-16:** *DONE* [openqa.opensuse.org infrastructure overview](https://youtu.be/G5bQKI2tURk) (see question in #88831#note-19 , @okurz)
* **2021-04-23:** *DONE* [openQA tests written in Python](https://youtu.be/GjKZ51lnCh0) (@okurz, @cdywan)
* **2021-04-30:** *DONE* [openqa-review: A review helper script for openQA with complete test overview reports](https://youtu.be/J2eI0gKnQNM) (@okurz)
* **2021-05-07:** *DONE* [SUSE QE Tools roadmap - 2021-05](https://youtu.be/J2eI0gKnQNM) (@okurz, @cdywan)
* **2021-05-14:** *DONE* [Review badges and recent changes related to them](https://youtu.be/rduc1z1HB-4) (@mkittler)
* **2021-05-21:** *DONE* [openQA API Playground](https://youtu.be/EfXZKbQS-Kg) (@okurz)
* **2021-05-28:** *DONE* [Tumbleweed workflows focussed on openQA](https://youtu.be/YiiuNqRPGAk) (proposal by okurz motivated by https://chat.suse.de/channel/testing?msg=EysbgG5kFrHbmjvcy , e.g. impact of failing tests, to-test manager, etc.; by okurz, dimstar?)*
* **2021-06-04:** *DONE* SUSE QE Tools roadmap - 2021-06
* **2021-06-11:** *DONE* [intro to os-autoinst development (demo how to investigate and test a small fix)](https://youtu.be/IeXaVb5dqy8) (@okurz, @mkittler)
* **2021-06-18:** *DONE* [How to be prepared when we introduce openQA features](https://youtu.be/wERuChD-88Y) (@cdywan, @okurz)
* **2021-06-25:** *DONE* Workflow discussions: SUSE QE aggregate tests (Proposed by okurz: We would like to learn from others how these are included in the workflow; no recording) (@okurz)
* **2021-07-02:** *DONE* [SUSE QE Tools roadmap - 2021-07](https://youtu.be/bppQFEhlfic) (@okurz, @cdywan)
* **2021-07-09:** *DONE* A glimpse into the QE Core workflow (@geor, @ilausuch)
* **2021-07-16:** *DONE* Testing SLES+HA & SAP Clusters with openQA (@acarvajal)
* **2021-07-23:** *DONE* [Sporadic failures](https://youtu.be/TB-QO3Ipo1E) (@punkioudi)
* **2021-07-30:** *DONE* [An Overview of the HANA Performance Continuous Integration](https://youtu.be/R4f4Lxr0-zk) (@jgwang)
* **2021-08-06:** *DONE* [SUSE QE Tools roadmap - 2021-08](https://youtu.be/6SVV3Kb0lSI) (@tinita, @cdywan)
* **2021-08-13:** *DONE* A proposal to maintain testsuites through a [GitLab CI pipeline](https://gitlab.suse.de/qa-maintenance/qam-openqa-yml/-/merge_requests/163) @apappas
* **2021-08-20:** *DONE* [Space management](https://youtu.be/g331EIPd_jQ) (@mkittler)
* **2021-08-27:** *DONE* [openQA soft-fails (what are they, how to use, when to use, limitations)](https://youtu.be/HZAvYw86-lw) (@okurz)
* **2021-09-03:** *DONE* [SUSE QE Tools roadmap - 2021-09](https://youtu.be/5o6hUkEfrsA) (@okurz)
* **2021-09-10:** *DONE* open conversation (@okurz)
* **2021-09-17:** *DONE* [discussing new openQA features (openqa-review, priorities, module search)](https://youtu.be/0-QiVh1qBbI) (@okurz)
* **2021-09-24:** *DONE* [Scripting openQA from top to bottom by phoenix](https://youtu.be/RUVtn6unMfs) (@cdywan, Felix)
* **2021-10-01:** *DONE* [SUSE QE Tools roadmap - 2021-10](https://youtu.be/n75pvgPO2L0) (@okurz)
* **2021-10-08:** *DONE* [Brainstorming and ideas for better connecting OBS+openQA (outgoing webhooks)](https://youtu.be/iRkZiNVSeKM) (@okurz, @hennevogel)
* **2021-10-15:** *DONE* [lessons learned from "Published QCOW images appear to be uncompressed" - #99579](https://youtu.be/DYe3C1HFHDQ) (@okurz)
* **2021-10-22:** *DONE* [Follow up to Testing SLES+HA & SAP Clusters with openQA: Test Results & Known Issues](https://youtu.be/nAVg_COmE5M) (@acarvajal, @okurz)
* **2021-10-29:** *DONE* [Override openQA job results with special comments](https://youtu.be/Ki_G6kR0c-o) (@okurz)
* **2021-11-05:** *DONE* [SUSE QE Tools roadmap - 2021-11](https://youtu.be/VkDkw2cE1Gg) (@mkittler)
* **2021-11-12:** *DONE* Explore integration of other tools, test frameworks, Integration, e.g. Selenium based tests in openQA (recording n/a) (@okurz, @ybonatakis)
* **2021-11-19:** *DONE* [Discussion & brainstorming: terraform provider for openQA (to manage machine, test suites, job groups, etc., essentially everything that lives in the database)](https://youtu.be/i1FvOishUEw) (@dancermak)
* **2021-11-26:** *DONE* [QA best practices (brainstorming session, proposal by ybonatakis)](https://youtu.be/NDD1Ku2FL04) (@okurz)
* **2021-12-03:** *DONE* [SUSE QE Tools roadmap - 2021-12](https://youtu.be/pfhXDPSCmjk) (@okurz)
* **2021-12-10:** *DONE* open conversation - Investigating failed openQA jobs, how to influence project decisions, how to learn about upcoming changes (no recording) (@cdywan)
* **2021-12-17:** *DONE* [mergify - automatic merges of pull requests and more](https://youtu.be/hhfmQCfP5tc) (@okurz)
* **2021-12-24:** *skipped due to holiday*
* **2021-12-31:** *skipped due to holiday*
* **2022-01-07:** *skipped due to holiday*
* **2022-01-14:** *open conversation* (@cdywan)
* **2022-01-21:** *DONE* [One year of SUSE QE Tools Workshop! Let's celebrate the success and have a good plan for the future](https://youtu.be/bzcEF5VAM7w) (@okurz)
* **2022-01-28:** *auto-review with force-result* (@tinita)
* **2022-02-04:** SUSE QE Tools roadmap - 2022-02 (@okurz)
* *proposal by okurz: How we review openQA test results, by SUSE QE teams: Who volunteers from each team to present? Propose a speaker and a date!*
* *proposal by okurz: openQA test review best practices and recent related feature development*
* *SMELT showcase (@vanastasiadis)*
* *proposal by okurz: Scaling up: openQA result archiving and more (#64746) (@mkittler, maybe start of 2022?)*
* *proposal by okurz: openQA feature: Retry of jobs based on test variables (#104007) (@okurz)*
* *proposal by okurz: The "TODO"-functionality on openQA (#93246)*
* *periodic proposal by okurz: How to report tickets, investigate issues, etc. (#104805)*
* *general proposal: if there are no further topics make it an "open conversation", at least from time to time :)*
* feedback from yearly workshop review: run it every second week but maybe longer, more interactive, more technical sessions, about backends and more openQA internals, from jlausuch: maybe understanding how svirt backend boots VMs in s390x, VMWare, etc? Highlight the differences between how qemu backend spawns VMs and how others do
#### Announcements
- For every meeting, regular or one-off, desired attendants should be invited to make sure a slot blocked in their calendar and reminders with the correct local time will show up when it's time to join the meeting
- Create a new event, for example in Thunderbird via the *Calendar* tab or `New > Event` via the menu.
- Pick your audience, for example `qa-team@suse.de` will reach test developers and reviewers, or you can select individual attendants via their respective email addresses.
- Add attendees accordingly.
- Specify the time of the meeting
- Set a schedule to repeat the event if applicable.
- Add a location, e.g. https://meet.opensuse.org/suse_qa_tools
- Don't worry if any of the details might change - you can update the invitation later and participants will be notified.
- See the respective meeting for regular actions such as communication via chat
### Team
The team is comprised of engineers from different teams, some only partially available:
* Cris Dywan (Scrum Master) @cdywan / [@kalikiana](https://github.com/kalikiana)
* Oliver Kurz (Product Owner)
* Marius Kittler
* Nick Singer (only OPS)
* Sebastian Riedel (Part time, **no alert duty**) @kraih / [@kraih](https://github.com/kraih)
* Tina Müller (Part time (35h)) @tinita / [@perlpunk](https://github.com/perlpunk)
* Jan Baier (part time, QEM-dedicated work areas)
* Ondřej Súkup (dedicated work areas) @osukup / [@mimi1vx](https://github.com/mimi1vx)
* Moritz Kodytek @kodymo / [@FruitFly638](https://github.com/FruitFly638)
### Onboarding for new joiners
* Request to get added to the [tools team on GitHub](https://github.com/orgs/os-autoinst/teams/tools-team) and subscribe to notifications for projects within that organization
* Subscribe to notifications of the [Mojo-IOLoop-ReadWriteProcess project on GitHub](https://github.com/openSUSE/Mojo-IOLoop-ReadWriteProcess) as it is also closely related to openQA development
* Login at [stats.openqa-monitor.qa.suse.de](https://stats.openqa-monitor.qa.suse.de/alerting/list) with NIS/LDAP credentials and ask to be given the *admin* role
* Watch this wiki page (click "Watch" button on top of this page)
* Subscribe to [osd-admins@suse.de](http://mailman.suse.de/mailman/listinfo/osd-admins), [openqa@suse.de](http://mailman.suse.de/mailman/listinfo/openqa) and [opensuse-factory@opensuse.org](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org)
* Join #suse-qe-tools:opensuse.org (matrix) and [team-qa-tools on Slack](https://suse.slack.com/archives/C02AJ1E568M)
* Request to join [devel:openQA on OBS](https://build.opensuse.org/project/show/devel:openQA) and check that you have `Request created`, `New comment for request created`, `New comment for project created`, `New comment for package created` enabled for `Maintainer of the target` in your [OBS notification settings](https://build.opensuse.org/my/subscriptions) (staging bot writes reminder comments on open reviews)
* Add [devel:openQA on OBS](https://build.opensuse.org/project/show/devel:openQA) to your watchlist
* Connect to `#opensuse-factory` on *libera.chat*, see "Common tasks for team members - Monitor" above
* Request admin access on [osd](http://openqa.suse.de/) and [o3](http://openqa.opensuse.org/)
* Request to get added to the [QA project in Progress](https://progress.opensuse.org/projects/qa/settings/members) and *enable notifications for the openQA project* in [your account settings](https://progress.opensuse.org/my/account)
* Request to get added to the [openqa team in GitLab](https://gitlab.suse.de/groups/openqa/-/group_members)
* Add your ssh key to https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/sshd/users.sls with a merge request
* Add your ssh key to gitlab.suse.de/qa-maintenance/qamops/-/blob/master/ansible/books/vars/main.yml with a merge request
* Ask an existing admin, e.g. other members of the team, to add your username and ssh key to o3, see https://progress.opensuse.org/projects/openqav3/wiki/#SSH-configuration
* Ensure you are subscribed to all projects referenced in https://progress.opensuse.org/projects/qa/wiki#Common-tasks-for-team-members
* ~~Ensure you have access to https://gitlab.suse.de/OPS-Service/monitoring (create EngInfra ticket otherwise) and add yourself in https://gitlab.suse.de/OPS-Service/monitoring/-/tree/master/icinga/shared/contacts to receive monitoring information~~ EngInfra does not grant access to additional people currently. That might change again in the future.
* Ask for access to the vacations calendar (on demand, via invitation)
* Ask for access to the meetings calendar (on demand, via invitation)
* *Watch* [qa-tools-backlog-assistant](https://github.com/os-autoinst/qa-tools-backlog-assistant) and choose *All Activity*
* Ensure you can access thruk.suse.de via NIS/LDAP credentials
### Alert handling
#### Best practices
* "if it hurts, do it more often": https://www.martinfowler.com/bliki/FrequencyReducesDifficulty.html
* Reduce [Mean-time-to-Detect (MTTD)](https://searchitoperations.techtarget.com/definition/mean-time-to-detect-MTTD) and [Mean-time-to-Recovery](https://raygun.com/blog/what-is-mttr/)
#### Process
* React on any alert or report of an outage
* If users report outages of components of our infrastructure
* Consider forming a task force and work together
* Inform the affected users about the impact, mitigation/workarounds and ETA for resolution
* For each failing alert, e.g. grafana
* Create a ticket for the issue (with a tag "alert"; create ticket unless the alert is trivial to resolve and needs no improvement; if an alert is unhandled for at least 4h then a ticket must be created; even create a ticket if alerts turn to "ok" to prevent these issues in the future and to improve the alter)
* Link the corresponding grafana panel in the ticket
* Respond to the notification email with a link to the ticket or forward the email to a corresponding mailing list, e.g. o3-admins@suse.de or osd-admins@suse.de (Caveat: gitlab@suse.de as sender seems to be able to receive emails and swallow them without any useful response or error message)
* Optional: Inform in chat
* Optional: Add "annotation" in corresponding grafana panel with a link to the corresponding ticket
* Pause the alert if you think further alerting the team does not help (e.g. you can work on fixing the problem, alert is non-critical but problem can not be fixed within minutes)
* If you consider an alert non-actionable then change it accordingly
* If you do not know how to handle an alert ask the team for help
* We must always strive for an accepted hypothesis when we want to change alerts or call an issue resolved
* After resolving the issue add explanation in ticket, unpause alert and verify it going to "ok" again, resolve ticket
#### References
* https://nl.devoteam.com/en/blog-post/monitoring-reduce-mean-time-recovery-mttr/
#### Gitlab Pipeline Notifications
Currently, the following projects are configured to write an email to osd-admins@suse.de if a pipeline fails:
* [openqa/auto-review](https://gitlab.suse.de/openqa/auto-review/-/services/pipelines_email/edit)
* [openqa/grafana-webhook-actions](https://gitlab.suse.de/openqa/grafana-webhook-actions/-/services/pipelines_email/edit)
* [openqa/monitor-o3](https://gitlab.suse.de/openqa/monitor-o3/-/services/pipelines_email/edit)
* [openqa/openqa-review](https://gitlab.suse.de/openqa/openqa-review/-/services/pipelines_email/edit)
* [openqa/osd-deployment](https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit)
* [openqa/salt-states-openqa](https://gitlab.suse.de/openqa/salt-states-openqa/-/services/pipelines_email/edit)
* [openqa/salt-pillars-openqa](https://gitlab.suse.de/openqa/salt-pillars-openqa/-/services/pipelines_email/edit)
* [qa-maintenance/bot-ng](https://gitlab.suse.de/qa-maintenance/bot-ng/-/services/pipelines_email/edit)
* [qa-maintenance/openQABot](https://gitlab.suse.de/qa-maintenance/openQABot/-/services/pipelines_email/edit)
- The configuration can be found by going to **Settings** > **Integrations** > **Pipeline Status Emails** (for any new projects the plugin will need to be enabled first)
- There's no way to subscribe as a user - instead an email address must be added
#### Weekly alert duty
We all should react on alert but additionally we can have one person on "alert duty" for one week each to ensure quicker reaction times when other team members are focussed on development work. For this the person on duty should do the following:
* React quickly (e.g. within some minutes or at latest hours) on any unhandled alerts
* Hand over to the next person on the beginning of the week, going by the order of team members in the wiki
* Asks for standin on unavailabilities
### Things to try
* Everybody can be "Product Owner" or "Scrum Master" or "Admin" or "Developer" for some time to get the different perspective
* From time to time ask stakeholders for their list of priorities regarding our tasks
### Literature references
* https://xahteiwi.eu/resources/presentations/no-we-wont-have-a-video-call-for-that/
### Extra-ordinary "hack-week" 2020-W51
SUSE QE Tools plans to have an internal "hack-week": Condition: We close 30 tickets from our backlog within the time frame 2020-12-03 until 2020-12-11 start of weekly meeting. No cheating! :) See [this query](https://progress.opensuse.org/issues?utf8=%E2%9C%93&set_filter=1&sort=priority%3Adesc%2Cid%3Adesc&f%5B%5D=status_id&op%5Bstatus_id%5D=c&f%5B%5D=fixed_version_id&op%5Bfixed_version_id%5D=%3D&v%5Bfixed_version_id%5D%5B%5D=418&f%5B%5D=closed_on&op%5Bclosed_on%5D=%3E%3C&v%5Bclosed_on%5D%5B%5D=2020-12-03&v%5Bclosed_on%5D%5B%5D=2020-12-11&f%5B%5D=&c%5B%5D=subject&c%5B%5D=project&c%5B%5D=status&c%5B%5D=assigned_to&c%5B%5D=relations&c%5B%5D=priority&c%5B%5D=category&c%5B%5D=cf_16&group_by=status&t%5B%5D=). During week 2020-W51 everyone is allowed to work on any hack-week project, it should just have a reasonable, "explainable" connection to our normal work. okurz volunteers to take over ops-duty for the week.
Result during meeting 2020-12-11: We missed the goal (by a slight amount) but we are motivated to try again in the next year :) Everybody, put some easy tickets aside for the next time!
### Extra-ordinary "hack-week" 2021-W8
Similar as our attempt for 2020-W51 with same rules, except condition: We close 30 tickets from our backlog within the time frame 2021-02-05 until 2021-02-19 start of weekly meeting. No cheating! See [this query](https://progress.opensuse.org/issues?utf8=%E2%9C%93&set_filter=1&sort=priority%3Adesc%2Cid%3Adesc&f%5B%5D=status_id&op%5Bstatus_id%5D=c&f%5B%5D=fixed_version_id&op%5Bfixed_version_id%5D=%3D&v%5Bfixed_version_id%5D%5B%5D=418&f%5B%5D=closed_on&op%5Bclosed_on%5D=%3E%3C&v%5Bclosed_on%5D%5B%5D=2021-02-05&v%5Bclosed_on%5D%5B%5D=2021-02-19&f%5B%5D=&c%5B%5D=subject&c%5B%5D=project&c%5B%5D=status&c%5B%5D=assigned_to&c%5B%5D=relations&c%5B%5D=priority&c%5B%5D=category&c%5B%5D=cf_16&group_by=status&t%5B%5D=).
Result during meeting 2021-02-19: We missed the goal (25/30 tickets resolved) but again we are open to try again, maybe after next SUSE hack week.
### Historical
Previously the former QA tools team used target versions "Ready" (to be planned into individual milestone periods or sprints), "Current Sprint" and "Done". However the team never really did use proper time-limited sprints so the distinction was rather vague. After having tickets "Resolved" after some time the PO or someone else would also update the target version to "Done" to signal that the result has been reviewed. This was causing a lot of ticket update noise for not much value considering that the [Definition-of-Done](https://progress.opensuse.org/projects/openqav3/wiki/#ticket-workflow) when properly followed already has rather strict requirements on when something can be considered really "Resolved" hence the team eventually decided to not use the "Done" target version anymore. Since about 2019-05 (and since okurz is doing more backlog management) the team uses priorities more as well as the status "Workable" together with an explicit team member list for "What the team is working on" to better visualize what is making team members busy regardless of what was "officially" planned to be part of the team's work. So we closed the target version. On 2020-07-03 okurz subsequently closed "Current Sprint" as also this one was in most cases equivalent to just picking an assignee for a ticket or setting to "In Progress". We can just distinguish between "(no version)" meaning untriaged, "Ready" meaning tools team should consider picking up these issues and "future" meaning that there is no plan for this to be picked up. Everything else is defined by status and priority.
In 2020-10-27 we discussed together to find out the history of the team. We clarified that the team started out as a not well defined "Dev+Ops" team. "team responsibilities" have been mainly unchanged since at least beginning of 2019. We agreed that learning from users and production about our "Dev" contributions is good, so this part of "Ops" is responsibility of everyone.
Also see #73060 for more details about how the responsibilities were setup.
## Change announcements
For new, cool features or disruptive changes consider providing according notifications to our common userbase as well as potential future users, for example create post on opensuse-factory@opensuse.org , link to post on openqa@suse.de , invite for workshop, post on one.suse.com, #opensuse-factory (IRC) (irc://irc.libera.chat/opensuse-factory), [#testing (RC)](https://chat.suse.de/testing)
# QE Core and QE Yast - Team descriptions
Moved to [[Core-and-yast]]