coordination #155608
Updated by szarate 5 months ago
Background: I met with Sam Thursfield from Gnome during FOSDEM, and one of the things that popped up was that we needed (both openSUSE & Gnome) to collaborate better and to avoid duplicating the work that we do for the same actions, I asked Sam to do a writeup of things they would need from Gnome perspective, things they have developed, that could also help us. There are Ideas I was already exploring like the Replay feature in QEMU, and while going through this document, I confirm that for a lot of the Ideas we have in the backlog of either openQA or the openQA tests, they have the heart in the right place. Source: https://gitlab.gnome.org/GNOME/openqa-tests/-/wikis/GNOME-End-to-end-Testing-Initiative/openQA-Wishlist In no particular order... * The web API and command server API allows building custom tooling around openQA * The Web UI makes fixing mismatched needles very easy * All logs are available so debugging is usually practical * Ready-built containers make it easier to set up * The video.ogv for the test run is fantastic to have * Upstream maintainers always friendly and responsive * This type of testing just wasn't possible before. ### A rough wishlist for openQA and the testing initiative. In no particular order. Some items realistic and others not. - Collaboration with downstream distro QA teams - Not sure how we do this. But we are duplicating some effort at the moment. #### .1 Documented support for transient workers - GNOME doesn't capacity to maintain a fleet of dedicated openQA workers, instead we use our existing set of Gitlab runners and create transient openQA workers on those. - This is currently not entirely supported in openQA; for example developer mode can't be used, and we have to (ab)use 'worker_class' to ensure jobs are routed to the right worker (see https://gitlab.gnome.org/GNOME/openqa-tests/-/blob/master/utils/setup_worker.sh?ref_type=heads). - Goal: document this way of using openQA upstream and fix any rough edges #### .1 Branching in needles and tests. .1 We don't want the working tree of the needles repo to grow infinitely - We want to branch the testsuite and needles for every GNOME release, i.e the 'main' / 'master' branch tests the latest 'master' version of GNOME OS, while 'gnome-45' branch tests GNOME 45, etc. - openQA calls tests 'gnomeos-master-iso.x86_64', how do we swap 'master' for 'gnome46' etc? - How can we ensure the correct version of the needles are used in the web UI and test? #### .1 Improved local development flows - GNOME is developed by many commandline users. It seems like openQA is designed for "web-first" use, where internet connection is always available and tests run on remote infra. This isn't ideal for integrating into existing module development workflows. - The single-instance container provides one way to run tests locally - running a full instance of openQA + os-autoinst. This works but is heavyweight. You still couldn't really integrate it into a Makefile-style local build. - An attempt at solving this is ssam_openqa tool, which wraps just the os-autoinst container. Apart from pulling the container, it's a lightweight workflow. - .1 Documenting and versioning the command server API would make building this kind of tool easier. - The ideal would be a graphical desktop app, but this requires a big investment. - QAnvas is another tool, a mini needle editor in HTML+JS #### .1 Docs for the support functions in os-autoinst-distri-opensuse - When I need something like serial console support, I copy paste from os-autoinst-distri-opensuse until something works, without any idea what I am really doing. - This stuff is going to be different for each distro, but there must be a way to better share and document this infrastructure. #### .1 A way to update all changed needles. - Sometimes GNOME lands a change which affects appearance of all applications. We want to include app appearance in the app needles, to guard against accidental regressions in the UI. But we also want a way to say "this change was expected, update all the needles with the new appearance". - .1 Better audio testing. When adding screenreader tests we found that an 80% match rate would treat no audio output at all as "success", while 90% would treat a difference in timing as "failed". - .1 Support for testing complex gestures e.g. multitouch - .1 Avoid the one second delay in every assert_and_click and click_lastmatch call - .1 Show mouse cursor so we can test cursor size changes - This becomes more relevant as we start testing GNOME on Mobile. - .1 Better SMBIOS support. We currently manually build and inject an SMBIOS with a hack; SMBIOS is very powerful and perhaps we could improve support in isotovideo for injecting configuration files and kernel args via SMBIOS. #57 - .1 Support for running the web UI in openShift - .1 Support for running openQA tests in each app's CI pipeline - .1 Hardware testing for GNOME - .1 Record and replay the entire test using https://www.qemu.org/docs/master/system/replay.html