QA » openQA Project

It is true that one has generally internet access on OBS service runs. However, since we cannot use any existing services without extra effort and our current way of fetching assets is a bit involved this is not so easy to exploit.

There are multiple ways to approach this and none of them seem very straight forward:

1. We depend entirely on npm modules. Then we could probably use https://github.com/openSUSE/obs-service-node_modules like cockpit.
   1. Switching to webpack would probably help but doesn't solve all of it because likely not all of our dependencies are packaged as npm modules. In addition, when I remember correctly, our assetpack scripts did things that were not easily replicable with webpack when I tried so this would be a huge refactoring.
2. We write our own obs-service. It would just invoke our existing script for updating the cache.
   1. I'm not sure about the deployment. Probably we needed to create an rpm package (like the ones found on https://build.opensuse.org/project/show/openSUSE:Tools) and ask OBS admins to deploy it.
   2. The problem might be that our script depends on Mojolicious and various other Perl packages. In fact, we'd probably require almost all of openQA's normal dependencies as we basically just start the web application. Probably deploying an obs-service package which such a huge dependency trail is not acceptable for OBS admins and maybe not all dependencies are even resolvable on whatever system they're using. (If this was simple enough than @coolo would probably have already implemented it this way back in the day.)
3. Use an existing service which would probably mean parsing assetpack.def and generating a list of files to be downloaded from that in the format expected by an existing service:
   1. https://github.com/openSUSE/obs-service-download_files reads the URLs from the specfile itself which is probably not what we want.
   2. https://github.com/openSUSE/obs-service-download_url lacks a README so I we had to reverse engineer for details on it.
   3. We might not be in control of the directory structure where the downloaded files are put under but we need that as we need to download files where assetpack would expect them.
      1. We could of course move files around at build time (in the rpm %build section) but that would not be very nice.
   4. We have tons of individual files to be downloaded. It would probably not be nice to clutter OBS with that. Our current asset update script solves that by simply creating the archive cache.txz instead of adding the files to OBS individually. Not sure whether that would be possible using such a service.
4. We write our own obs-service. It would not invoke our existing script but be much simpler instead.
   1. It would only parse assetpack.def and download files where assetpack would expect them.
   2. To avoid cluttering OBS we could create the archive cache.txz as before.

I asked on the OBS channel what's the best approach is. Because IRC is not very persistent:

[14:37] Any idea what the best approach for caching external assets for openQA builds on OBS would be?
[14:38] I suppose it would boil down to making use on an OBS service of some kind. I already created a list of some options: https://progress.opensuse.org/issues/153427#note-6
[14:38] Unfortunately none of those options is particularly easy.
[14:41] I'm aware that other complex packages rely on the maintainer updating the package to run some script locally (like the Grafana package that comes with its own Makefile which in turn invokes yarn and tons of things I don't even want to know). That's what we did so far for openQA as well but it would be desirable to automate this as we generally build new packages as there are new changes in Git and we even have OBS
[14:41] package builds as CI checks.

I'm waiting for feedback. I was suggested to read https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.scm_bridge as it might not make sense anymore putting so much effort into the old handling of sources on OBS.

[15:56] in short, obs is just following the sources of a git server and is not doing any kind of modifications anymore
[15:56] so entire workflow needs to be eg. in gitea actions or external services. And entire source state of entire project can be verified via a single identifier
[15:57] there is also a remoteasset implementation, but most likely not helpful for you

That would mean we'd had some other job (e.g. a GitHub action) that pushes the assets we need on some Git branch that is then consumed by OBS (like it currently does for our normal source code or with the SCM bridge approach when we migrate to that). That in turn means:

1. We need to run a CI action on every PR which is potentially pushing changes on a newly created Git branch that is then configured to be used by the packaging CI checks.
   1. "potentially" because the action would likely only be triggered (or do anything) if assetpack.def was changed.
   2. The CI checks for packaging needed to wait until the action pushing changes has completed. This might not be easy to solve because the CI checks for packaging are simply triggered by a webhook. So we cannot make use of e.g. the dependency handling within CircleCI.
2. We need to run a CI action after every merged PR which is potentially pushing changes on a dedicated Git branch that is configured to be used by our packaging on devel:openQA.
   1. Since the Git branch the changes would be pushed on is just one for the whole repository we need to synchronize the CI actions (so there's no conflict when PRs are merged shortly after another).

That's definitely some work, or am I over-complicating things?

As a start we could just create an "assets" Git branch manually and configure our packaging to use it.

That's definitely some work, or am I over-complicating things?

I think you are not overcomplicating things if we really want to build and package openQA and the assets within OBS.

I would prefer to not maintain more branches. We already have master and gh-pages. Admittedly gh-pages is really low effort but still. How about we consider for a chance how one would build an openQA container from the git sources. How would that work? And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?

How about we consider for a chance how one would build an openQA container from the git sources. How would that work?

Regardless of using a container or not, one would have to download assets on the first launch of the web service - unless we also improve that. Having all files that would be downloaded directly in our on Git repo at the right place would of course improve that.

And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?

I would not create an archive then but simply add the files to Git where they would end up anyway. Otherwise we would add a binary file in Git and the user would have to extract the archive first (like the rpm build currently does). I personally don't have a problem with having those external files in our Git repo in general although we need to clarify the licensing¹.

Note that asset pack produces a different output for different modes (development/testing/ production). We also actually use this to have minimized files in production and to disable certain effects when testing (via Selenium). I suppose this is not going to be a problem as long as we really only check-in the files that would be downloaded. The build script could then invoke the web application once to generate the remaining files (so the final package is complete and doesn't rely on computing files anymore at startup which would probably also run into permission errors).

¹In a personal repository I actually did the same and just checked in the licensing files that came by installing the npm package anyway (e.g. https://github.com/Martchus/arch-repo-manager/tree/master/srv/static/node_modules/xterm). In our case we don't use npm but could just download and add the LICENSING files manually.

RFC-PR for adding assets directly to our Git repository: https://github.com/os-autoinst/openQA/pull/5454

RFC-PR for providing assets via npm: https://github.com/os-autoinst/openQA/pull/5463

Unfortunately this doesn't seem to cut it. The obs-service-node_modules service can apparently not be used that way. I only realize now that cockpit also only uses it with mode="manual" which means it doesn't actually run on OBS.
Besides that, it would also require more additional work to adapt the CI. I already put some effort into it but there are still problems.

Maybe we should reiterate the options we have (e.g. in the next unblock meeting).

It looks like the problem was fixable after all and the service runs now on my PR branch on OBS: https://build.opensuse.org/package/show/devel:openQA:GitHub:os-autoinst:openQA:PR-5463/openQA
The builds is still running as I write this comment but it looks promising. I'll download the rpm and test whether it actually works locally.

I was also able to adapt CI checks. There are still some fixes remaining as now the style check wants to go over files from the node_modules directory.

I also still need to figure out that handling of fonts in production mode (where the URLs are not correctly resolved).

• After my latest changes CI checks should work.
• I tested the RPMs locally and found issues. It should be fixed after my latest changes but I still need to check that once the RPMs have been rebuilt.
• The mentioned issue about resolving URLs could be fixed by amending the Fetch-pipeline of AssetPack.
• I linked the local-npm-registry package in our devel repos to get Leap builds going. I created https://build.opensuse.org/request/show/1146197 for SLE.
• I checked https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit used by our Tumbleweed submissions. It looks like the script will cope with the newly added service without modifications. (I haven't tested it yet. Maybe we need slight modifications after all but this shouldn't be a big deal.)

Sounds like a plan! I think we are at the point where after you fix remaining CI failures we can merge your PR which goes way beyond what I officially thought we could come up with improvements.

https://github.com/os-autoinst/openQA/pull/5463 merged and apparently already deployed to o3 as it caused a problem

https://suse.slack.com/archives/C02CANHLANP/p1707834084412189?thread_ts=1707834084.412189&cid=C02CANHLANP
Reports

 openqa-gru[7430]: Can't open file "/usr/share/openqa/script/../assets/../node_modules/datatables.net-bs4/css/dataTables.bootstrap4.min.css": Permission denied

Mitigated by fvogt

I did sudo aa-complain /usr/share/openqa/script/openqa and it works again.

Please add the according reports from logs to apparmor profiles. I will create a separate ticket to include apparmor in openQA-in-openQA tests.

This should fix the permission problem: https://github.com/os-autoinst/openQA/pull/5470

So I'm finally getting to hitting this in the Fedora package. In Fedora's build system we have no network access ever for any reason, so I cannot use the new approach. Should it be safe to keep using our existing asset cache script - https://src.fedoraproject.org/rpms/openqa/blob/rawhide/f/update-cache.sh - which basically calls tools/generate-packed-assets and then tars up assets/cache and assets/assetpack.db ? Is that still a valid approach with all the NPM and whatever changes? Thanks.

edit: looks like I just needed to make the cache script depend on npm, have it run npm install before tools/generate-packed-assets, and include node_modules in the tarball.