QA » openQA Project

I suggest to also read https://dev.to/lostintangent/10-awesome-things-you-can-do-with-github-dev-5fm7

I was able to build from this Dockerfile and run it in codespace:

FROM docker.io/opensuse/leap:15.6

RUN zypper in -y git bash

RUN zypper in -y openQA-single-instance openQA-bootstrap \
    qemu-arm qemu-ppc qemu-x86 qemu-tools sudo iputils os-autoinst-distri-opensuse-deps && \
    zypper clean -a

Branch: https://github.com/perlpunk/openQA/tree/codespace

I started openqa-bootstrap and had a working webui on https://some-name-80.app.github.dev.

I cloned this test: https://openqa.opensuse.org/tests/4191283
However the test died:

[2024-05-15T21:55:44.230858Z] [info] [pid:6527] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  QEMU terminated before QMP connection could be established. Check for errors below
[2024-05-15T21:55:44.231141Z] [info] [pid:6527] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2024-05-15T21:55:44.234059Z] [debug] [pid:6527] Passing remaining frames to the video encoder
[2024-05-15T21:55:44.334945Z] [debug] [pid:6527] Waiting for video encoder to finalize the video
[2024-05-15T21:55:44.335038Z] [debug] [pid:6527] The built-in video encoder (pid 6530) terminated
[2024-05-15T21:55:44.335922Z] [debug] [pid:6527] QEMU: QEMU emulator version 8.2.2 (SUSE Linux Enterprise 15)
[2024-05-15T21:55:44.335985Z] [debug] [pid:6527] QEMU: Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
[2024-05-15T21:55:44.336058Z] [warn] [pid:6527] !!! : qemu-system-x86_64: CPU model 'host' requires KVM or HVF
[2024-05-15T21:55:44.336762Z] [debug] [pid:6527] sending magic and exit
[2024-05-15T21:55:44.337059Z] [debug] [pid:6294] received magic close
[2024-05-15T21:55:44.441401Z] [warn] [pid:6294] !!! main: failed to start VM at /usr/lib/os-autoinst/backend/driver.pm line 104.

[2024-05-15T21:55:44.441738Z] [debug] [pid:6294] stopping command server 6516 because test execution ended through exception

I also cloned a different test using a Python test module and it complained that Inline::Python is not installed.

Adding "runArgs": [ "--privileged", "--device", "/dev/kvm" ] in devcontainer.json did not help

I added QEMU_APPEND="-enable-kvm" to openqa-clone-job. Now I get:

[2024-05-16T08:45:29.471397Z] [debug] [pid:6331] QEMU: Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers
[2024-05-16T08:45:29.471444Z] [debug] [pid:6331] QEMU: Could not access KVM kernel module: Permission denied
[2024-05-16T08:45:29.471548Z] [warn] [pid:6331] !!! : qemu-system-x86_64: failed to initialize kvm: Permission denied
[2024-05-16T08:45:29.472298Z] [debug] [pid:6331] sending magic and exit
[2024-05-16T08:45:29.472581Z] [debug] [pid:6098] received magic close
[2024-05-16T08:45:29.577241Z] [warn] [pid:6098] !!! main: failed to start VM at /usr/lib/os-autoinst/backend/driver.pm line 104.

I did:

chown root:kvm /dev/kvm

Now I get:

[2024-05-16T08:49:46.622010Z] [warn] [pid:8060] !!! : qemu-system-x86_64: -vnc :91,share=force-shared: Failed to find an available port: Address already in use

In our collab session we tried several things to change the vnc port.
In the end it helped to restart the worker instance with an instance number of 10 instead of 1. (we needed to create the pool directory first)
With that qemu started and the test passed.

We didn't find out why the default 5091 port did not work, as there was nothing else listening on that port.

But it would be good if the bootstrap script and our code was a bit more flexible to configure certain ports.
Also it would be nice if one could just stop everything and restart the bootstrap script after making changes. Running the bootstrap script twice currently fails early when adding the zypper repository (complaining that it already exists).

https://github.com/os-autoinst/openQA/pull/5644 Make VNC port offset configurable

Developer Tools reveals some more errors:

commons-app-axios.cj…f64113844588ba.js:1 Refused to set unsafe header "User-Agent"
371.a061e5c….js:1 vscs:web-client:ae39:codespaces-component:workbench-page Tunnel client is unconnected, fetching tunnel and connecting 
Object
commons-app-axios.cj…f64113844588ba.js:1 Refused to set unsafe header "User-Agent"
371.a061e5c….js:1 vscs:web-client:ae39:codespaces-component:workbench-page Created port https/16635 with privacy 'private'
371.a061e5c….js:1 vscs:web-client:ae39:codespaces-component:workbench-page:workbench:vscs:web-client:ae39:codespaces-component:workbench-page:workbench:vscode-workbench Failed to complete start-remote-server with gRPC exception was thrown by handler. exception: failed to start vs code remote server.
Error: exception was thrown by handler. exception: failed to start vs code remote server.
    at Object.onEnd (https://assets.github.dev/static/66.bbfdbcf….js:1:283263)
    at https://assets.github.dev/static/66.bbfdbcf….js:1:24222
    at https://assets.github.dev/static/66.bbfdbcf….js:1:11479
    at Array.forEach (<anonymous>)
    at e.rawOnError (https://assets.github.dev/static/66.bbfdbcf….js:1:11441)
    at e.onTransportHeaders (https://assets.github.dev/static/66.bbfdbcf….js:1:9014)
    at y.<anonymous> (https://assets.github.dev/static/app.3abbf6c….js:1:156890)
    at Generator.next (<anonymous>)
    at https://assets.github.dev/static/app.3abbf6c….js:1:154060
    at new Promise (<anonymous>)
371.a061e5c….js:1 vscs:web-client:ae39:codespaces-component:workbench-page:workbench:vscs:web-client:ae39:codespaces-component:workbench-page:workbench:vscode-workbench Failed to execute start-remote-server: undefined
371.a061e5c….js:1 vscs:web-client:ae39:codespace-state-watcher codespace state -> Available
371.a061e5c….js:1 vscs:web-client:ae39:workbench-telemetry:telemetry-service ["codespaces/web/get-connection-token-error",{"errorCode":-1,"errorType":"GenericError","errorMessage":"exception was thrown by handler. exception: failed to start vs code remote server.","originalErrorType":"[no original type]","errorDidCatch":false,"stack":"Error: exception was thrown by handler. exception: failed to start vs code remote server.\n    at Object.onEnd [PATH]/66.bbfdbcf….js:1:283263)\n    at [PATH]/66.bbfdbcf….js:1:24222\n    at [PATH]/66.bbfdbcf….js:1:11479\n    at Array.forEach (<anonymous>)\n    at e.rawOnError [PATH]/66.bbfdbcf….js:1:11441)\n    at e.onTransportHeaders [PATH]/66.bbfdbcf….js:1:9014)\n    at y.<anonymous> [PATH]/app.3abbf6c….js:1:156890)\n    at Generator.next (<anonymous>)\n    at [PATH]/app.3abbf6c….js:1:154060\n    at new Promise (<anonymous>)"}]
log.ts:439   ERR Error received from starting extension host (kind: Remote)
log.ts:439   ERR Error: exception was thrown by handler. exception: failed to start vs code remote server.
    at Object.onEnd (66.bbfdbcf….js:1:283263)
    at 66.bbfdbcf….js:1:24222
    at 66.bbfdbcf….js:1:11479
    at Array.forEach (<anonymous>)
    at e.rawOnError (66.bbfdbcf….js:1:11441)
    at e.onTransportHeaders (66.bbfdbcf….js:1:9014)
    at y.<anonymous> (app.3abbf6c….js:1:156890)
    at Generator.next (<anonymous>)
    at app.3abbf6c….js:1:154060
    at new Promise (<anonymous>)
...
But some of them I also see when it's working.

Just to recap the points from the unblock:

• Switching to Leap for now would make things work
• Issues with Tumbleweed are not currently reproducible outside of Codespaces
• Moving the existing container to leap, or making a new one (whichever is easier) so an image can be specified to avoid the slow re-builds for every person using it
• Codespaces as root works fine, would be nice if that can be a user, but could probably be left as-is as well

I'm trying out the single-instance-container with Leap 15.6. Codespace runs into the same error message using registry.opensuse.org/home/tinita/single-instance-openqa/containers/opensuse/openqa-single-instance:latest , so I'm commenting out everything and start commenting in step by step to see what's the problem.
It's a bit annoying, as a repeated click on "Trigger services" doesn't fetch new sources. It only works once.
So I have to create a new branch in git everytime and change the url in the _service file.
Maybe time for a bug report for OBS.

I'm making very slow progress.

I'm now trying out Leap 1.5, because 15.6 did not work:
During building in OBS, it has a newer version of qemu available than later when the container is running. This seems to be special to Leap 15.6.

Now whenever I want to try out the container in codespaces and call the bootstrap script I get problems with download.pensuse.org:

Signature verification failed for file 'repomd.xml' from repository 'Update repository of openSUSE Backports'. Continue? [yes/no] (no): no
Retrieving repository 'Update repository of openSUSE Backports' metadata ................[error]
Repository 'Update repository of openSUSE Backports' is invalid.
[repo-backports-update|http://download.opensuse.org/update/leap/15.5/backports/] Valid metadata not found at specified URL
History:
 - Signature verification failed for repomd.xml

This is getting annoying and I don't know how to proceed currently.
I can try and to something else and regularly check again if it is working, but I would finally like to concentrate on doing something for a longer time.

Now after writing this I tried again and download.o.o was fine again.

What's also annoying is that this costs time and storage on Github. There is a certain amount of CPU time and storage included, but if I keep a container running in the background only because download.o.o is unstable and I need to retry regularly my included time and storage will be exceeding its limit at some point.
I have ~13 hours left now for the next 11 days.

https://github.com/os-autoinst/openQA/pull/5664 Downgrade single-instance-container to Leap
I had to trigger services manually here:
https://build.opensuse.org/package/show/devel:openQA/openQA-single-instance-container
Now the source has been updated and it shows as unresolvable.
Need to figure out what to do.

x86_64
 unresolvable
The build can not begin, because required packages are either missing or not explicitly defined.

 Repository has been published
unresolvable: nothing provides container:opensuse/leap:15.5

I changed the Meta for the package and added <enable repository="containers15.5"/>. Now it built.
registry.opensuse.org/devel/openqa/containers15.5/openqa-single-instance:latest should be the new container url.

https://github.com/os-autoinst/openQA/pull/5648

Please also see https://github.com/os-autoinst/openQA/pull/5664/files#r1623190048 about the container versions. If we have problems with specific container versions but depending on the environment then we can also try to build different versions in parallel but keep Tumbleweed the default.

okurz wrote in #note-26:

Please also see https://github.com/os-autoinst/openQA/pull/5664/files#r1623190048 about the container versions. If we have problems with specific container versions but depending on the environment then we can also try to build different versions in parallel but keep Tumbleweed the default.

This was already discussed at length with multiple team members. I think Tina did or wanted to file a follow-up ticket on the container issues. Let's not go in circles here.

I created #161399 and https://github.com/orgs/community/discussions/127083 to track the Codespaces + Tumbleweed bug.

https://github.com/os-autoinst/openQA/pull/5671 Add documentation for codespaces

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/176 Add devcontainer for Codespaces
https://github.com/os-autoinst/os-autoinst-distri-example/pull/41 Add devcontainer for Codespaces

Personal idea is that Codespace is about having a preconfigured IDE to contribute to the repo directly from within the browser. With this personal point of view I've given a try to Codespace in https://github.com/os-autoinst/openQA

I create one, wait for the creation, then opened a second terminal to run tail -f /var/log/openqa-bootstrap.log and wait.

Then I opened in the editor the file https://github.com/os-autoinst/openQA/blob/master/docs/Contributing.asciidoc and read through it. One of the first indication there is about how to tidy the code. So I opened a terminal in the codespace and typed make tidy.

What I get is

codespaces-3ad963:/workspaces/openQA # make  tidy
bash: make: command not found

What I's suggesting is to have more tools installed in the image, not only to be able to trigger test in openQA but also to navigate and write code following the suggestions from the contribute document.

P.S. Some more experiments

codespaces-3ad963:/workspaces/openQA #  zypper in make nodejs npm

...


codespaces-3ad963:/workspaces/openQA #  make tidy

...
tools/tidy
Wrong version of perltidy. Found '20230309', expected '20240511'. Consider '--force' but results might not be consistent.
make: *** [Makefile:325: tidy-perl] Error 1

Using zypper in openQA-devel, as suggested in the contribute doc, does not solve this particular perltidy issue

https://github.com/os-autoinst/os-autoinst-distri-example/pull/43 codespace: Run chown as postStartCommand