openQA Project (public)

Background: I met with Sam Thursfield from Gnome during FOSDEM, and one of the things that popped up was that we needed (both openSUSE & Gnome) to collaborate better and to avoid duplicating the work that we do for the same actions, I asked Sam to do a writeup of things they would need from Gnome perspective, things they have developed, that could also help us.

There are Ideas I was already exploring like the Replay feature in QEMU, and while going through this document, I confirm that for a lot of the Ideas we have in the backlog of either openQA or the openQA tests, they have the heart in the right place.

Source: https://gitlab.gnome.org/GNOME/openqa-tests/-/wikis/GNOME-End-to-end-Testing-Initiative/openQA-Wishlist

In no particular order...

• The web API and command server API allows building custom tooling around openQA
• The Web UI makes fixing mismatched needles very easy
• All logs are available so debugging is usually practical
• Ready-built containers make it easier to set up
• The video.ogv for the test run is fantastic to have
• Upstream maintainers always friendly and responsive
• This type of testing just wasn't possible before.

A rough wishlist for openQA and the testing initiative.¶

In no particular order. Some items realistic and others not.

• Collaboration with downstream distro QA teams
  • Not sure how we do this. But we are duplicating some effort at the moment.

Documented support for transient workers¶

• GNOME doesn't capacity to maintain a fleet of dedicated openQA workers, instead we use our existing set of Gitlab runners and create transient openQA workers on those.
• This is currently not entirely supported in openQA; for example developer mode can't be used, and we have to (ab)use 'worker_class' to ensure jobs are routed to the right worker (see https://gitlab.gnome.org/GNOME/openqa-tests/-/blob/master/utils/setup_worker.sh?ref_type=heads).
• Goal: document this way of using openQA upstream and fix any rough edges

Branching in needles and tests.¶

.1 We don't want the working tree of the needles repo to grow infinitely

• We want to branch the testsuite and needles for every GNOME release, i.e the 'main' / 'master' branch tests the latest 'master' version of GNOME OS, while 'gnome-45' branch tests GNOME 45, etc.
• openQA calls tests 'gnomeos-master-iso.x86_64', how do we swap 'master' for 'gnome46' etc?
• How can we ensure the correct version of the needles are used in the web UI and test?

Improved local development flows¶

• GNOME is developed by many commandline users. It seems like openQA is designed for "web-first" use, where internet connection is always available and tests run on remote infra. This isn't ideal for integrating into existing module development workflows.
• The single-instance container provides one way to run tests locally - running a full instance of openQA + os-autoinst. This works but is heavyweight. You still couldn't really integrate it into a Makefile-style local build.
• An attempt at solving this is ssam_openqa tool, which wraps just the os-autoinst container. Apart from pulling the container, it's a lightweight workflow.
• Documenting and versioning the command server API would make building this kind of tool easier.
  • The ideal would be a graphical desktop app, but this requires a big investment.
• QAnvas is another tool, a mini needle editor in HTML+JS

Docs for the support functions in os-autoinst-distri-opensuse¶

• When I need something like serial console support, I copy paste from os-autoinst-distri-opensuse until something works, without any idea what I am really doing.
• This stuff is going to be different for each distro, but there must be a way to better share and document this infrastructure.

A way to update all changed needles.¶

• Sometimes GNOME lands a change which affects appearance of all applications. We want to include app appearance in the app needles, to guard against accidental regressions in the UI. But we also want a way to say "this change was expected, update all the needles with the new appearance".
• Better audio testing. When adding screenreader tests we found that an 80% match rate would treat no audio output at all as "success", while 90% would treat a difference in timing as "failed".
• Support for testing complex gestures e.g. multitouch
• Avoid the one second delay in every assert_and_click and click_lastmatch call

• Show mouse cursor so we can test cursor size changes

  • This becomes more relevant as we start testing GNOME on Mobile.

• Better SMBIOS support. We currently manually build and inject an SMBIOS with a hack; SMBIOS is very powerful and perhaps we could improve support in isotovideo for injecting configuration files and kernel args via SMBIOS. #57

• Support for running the web UI in openShift

• Support for running openQA tests in each app's CI pipeline

• Hardware testing for GNOME

• Record and replay the entire test using https://www.qemu.org/docs/master/system/replay.html