action #153427
closedcoordination #154768: [saga][epic][ux] State-of-art user experience for openQA
coordination #157510: [epic] Up-to-date JavaScript stack
Improve updating cached assets size:M
0%
Description
Motivation¶
In #153145 we saw that the dist/rpm/update-cache.sh
script does not work in the case when we add new assets or change urls for existing assets.
If you make a pull request with a new url, then it will fail in OBS because it has the old cache.
But running update-cache.sh will use the upstream openQA git repo which has the old url.
So it's a chicken-egg problem.
That also is related to when others try to build openQA outside the context of OBS, see recent discussions in https://app.element.io/#/room/#openqa:opensuse.org
Acceptance criteria¶
- AC1: It is not necessary to manually update the cached assets for rpm packaging when a remotly fetched asset is changed (in
assetpack.def
)
Suggestions¶
- Read and review http://open.qa/docs/#update-asset-cache
- Research or ask other users of OBS for best practices of caching external assets as OBS can not directly download external packages
- Consider using OBS services to download/update assets
- https://github.com/openSUSE/obs-service-download_files ?
- The cockpit package requires a ton of npm packages to be bundled, maybe a good example (https://build.opensuse.org/package/show/openSUSE:Factory/cockpit)
- Try it out with changing/adding assets like a new javascript module pulled in in a temporary draft PR, e.g. jquery upgrade?
- If we'd use webpack we could reuse the approach taken by cockpit packaging. Revisit #71476
Updated by tinita 11 months ago
- Copied from action #153145: [ci] Failing openQA pull requests in generate-packed-assets step added
Updated by mkittler 10 months ago · Edited
It is true that one has generally internet access on OBS service runs. However, since we cannot use any existing services without extra effort and our current way of fetching assets is a bit involved this is not so easy to exploit.
There are multiple ways to approach this and none of them seem very straight forward:
- We depend entirely on npm modules. Then we could probably use https://github.com/openSUSE/obs-service-node_modules like cockpit.
- Switching to webpack would probably help but doesn't solve all of it because likely not all of our dependencies are packaged as npm modules. In addition, when I remember correctly, our assetpack scripts did things that were not easily replicable with webpack when I tried so this would be a huge refactoring.
- We write our own obs-service. It would just invoke our existing script for updating the cache.
- I'm not sure about the deployment. Probably we needed to create an rpm package (like the ones found on https://build.opensuse.org/project/show/openSUSE:Tools) and ask OBS admins to deploy it.
- The problem might be that our script depends on Mojolicious and various other Perl packages. In fact, we'd probably require almost all of openQA's normal dependencies as we basically just start the web application. Probably deploying an obs-service package which such a huge dependency trail is not acceptable for OBS admins and maybe not all dependencies are even resolvable on whatever system they're using. (If this was simple enough than @coolo would probably have already implemented it this way back in the day.)
- Use an existing service which would probably mean parsing
assetpack.def
and generating a list of files to be downloaded from that in the format expected by an existing service:- https://github.com/openSUSE/obs-service-download_files reads the URLs from the specfile itself which is probably not what we want.
- https://github.com/openSUSE/obs-service-download_url lacks a README so I we had to reverse engineer for details on it.
- We might not be in control of the directory structure where the downloaded files are put under but we need that as we need to download files where assetpack would expect them.
- We could of course move files around at build time (in the rpm
%build
section) but that would not be very nice.
- We could of course move files around at build time (in the rpm
- We have tons of individual files to be downloaded. It would probably not be nice to clutter OBS with that. Our current asset update script solves that by simply creating the archive
cache.txz
instead of adding the files to OBS individually. Not sure whether that would be possible using such a service.
- We write our own obs-service. It would not invoke our existing script but be much simpler instead.
- It would only parse
assetpack.def
and download files where assetpack would expect them. - To avoid cluttering OBS we could create the archive
cache.txz
as before.
- It would only parse
I asked on the OBS channel what's the best approach is. Because IRC is not very persistent:
[14:37] Any idea what the best approach for caching external assets for openQA builds on OBS would be?
[14:38] I suppose it would boil down to making use on an OBS service of some kind. I already created a list of some options: https://progress.opensuse.org/issues/153427#note-6
[14:38] Unfortunately none of those options is particularly easy.
[14:41] I'm aware that other complex packages rely on the maintainer updating the package to run some script locally (like the Grafana package that comes with its own Makefile which in turn invokes yarn and tons of things I don't even want to know). That's what we did so far for openQA as well but it would be desirable to automate this as we generally build new packages as there are new changes in Git and we even have OBS
[14:41] package builds as CI checks.
I'm waiting for feedback. I was suggested to read https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.scm_bridge as it might not make sense anymore putting so much effort into the old handling of sources on OBS.
Updated by mkittler 10 months ago
[15:56] in short, obs is just following the sources of a git server and is not doing any kind of modifications anymore
[15:56] so entire workflow needs to be eg. in gitea actions or external services. And entire source state of entire project can be verified via a single identifier
[15:57] there is also a remoteasset implementation, but most likely not helpful for you
That would mean we'd had some other job (e.g. a GitHub action) that pushes the assets we need on some Git branch that is then consumed by OBS (like it currently does for our normal source code or with the SCM bridge approach when we migrate to that). That in turn means:
- We need to run a CI action on every PR which is potentially pushing changes on a newly created Git branch that is then configured to be used by the packaging CI checks.
- "potentially" because the action would likely only be triggered (or do anything) if
assetpack.def
was changed. - The CI checks for packaging needed to wait until the action pushing changes has completed. This might not be easy to solve because the CI checks for packaging are simply triggered by a webhook. So we cannot make use of e.g. the dependency handling within CircleCI.
- "potentially" because the action would likely only be triggered (or do anything) if
- We need to run a CI action after every merged PR which is potentially pushing changes on a dedicated Git branch that is configured to be used by our packaging on
devel:openQA
.- Since the Git branch the changes would be pushed on is just one for the whole repository we need to synchronize the CI actions (so there's no conflict when PRs are merged shortly after another).
That's definitely some work, or am I over-complicating things?
As a start we could just create an "assets" Git branch manually and configure our packaging to use it.
Updated by okurz 10 months ago
That's definitely some work, or am I over-complicating things?
I think you are not overcomplicating things if we really want to build and package openQA and the assets within OBS.
I would prefer to not maintain more branches. We already have master and gh-pages. Admittedly gh-pages is really low effort but still. How about we consider for a chance how one would build an openQA container from the git sources. How would that work? And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?
Updated by mkittler 10 months ago · Edited
How about we consider for a chance how one would build an openQA container from the git sources. How would that work?
Regardless of using a container or not, one would have to download assets on the first launch of the web service - unless we also improve that. Having all files that would be downloaded directly in our on Git repo at the right place would of course improve that.
And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?
I would not create an archive then but simply add the files to Git where they would end up anyway. Otherwise we would add a binary file in Git and the user would have to extract the archive first (like the rpm build currently does). I personally don't have a problem with having those external files in our Git repo in general although we need to clarify the licensing¹.
Note that asset pack produces a different output for different modes (development/testing/ production). We also actually use this to have minimized files in production and to disable certain effects when testing (via Selenium). I suppose this is not going to be a problem as long as we really only check-in the files that would be downloaded. The build script could then invoke the web application once to generate the remaining files (so the final package is complete and doesn't rely on computing files anymore at startup which would probably also run into permission errors).
¹In a personal repository I actually did the same and just checked in the licensing files that came by installing the npm package anyway (e.g. https://github.com/Martchus/arch-repo-manager/tree/master/srv/static/node_modules/xterm). In our case we don't use npm but could just download and add the LICENSING files manually.
Updated by mkittler 10 months ago
RFC-PR for adding assets directly to our Git repository: https://github.com/os-autoinst/openQA/pull/5454
Updated by mkittler 10 months ago · Edited
RFC-PR for providing assets via npm: https://github.com/os-autoinst/openQA/pull/5463
Unfortunately this doesn't seem to cut it. The obs-service-node_modules service can apparently not be used that way. I only realize now that cockpit also only uses it with mode="manual"
which means it doesn't actually run on OBS.
Besides that, it would also require more additional work to adapt the CI. I already put some effort into it but there are still problems.
Maybe we should reiterate the options we have (e.g. in the next unblock meeting).
Updated by mkittler 10 months ago · Edited
It looks like the problem was fixable after all and the service runs now on my PR branch on OBS: https://build.opensuse.org/package/show/devel:openQA:GitHub:os-autoinst:openQA:PR-5463/openQA
The builds is still running as I write this comment but it looks promising. I'll download the rpm and test whether it actually works locally.
I was also able to adapt CI checks. There are still some fixes remaining as now the style check wants to go over files from the node_modules
directory.
I also still need to figure out that handling of fonts in production mode (where the URLs are not correctly resolved).
Updated by mkittler 10 months ago
- After my latest changes CI checks should work.
- I tested the RPMs locally and found issues. It should be fixed after my latest changes but I still need to check that once the RPMs have been rebuilt.
- The mentioned issue about resolving URLs could be fixed by amending the
Fetch
-pipeline of AssetPack. - I linked the local-npm-registry package in our devel repos to get Leap builds going. I created https://build.opensuse.org/request/show/1146197 for SLE.
- I checked https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit used by our Tumbleweed submissions. It looks like the script will cope with the newly added service without modifications. (I haven't tested it yet. Maybe we need slight modifications after all but this shouldn't be a big deal.)
Updated by okurz 10 months ago
https://github.com/os-autoinst/openQA/pull/5463 merged and apparently already deployed to o3 as it caused a problem
openqa-gru[7430]: Can't open file "/usr/share/openqa/script/../assets/../node_modules/datatables.net-bs4/css/dataTables.bootstrap4.min.css": Permission denied
Mitigated by fvogt
I did sudo aa-complain /usr/share/openqa/script/openqa and it works again.
Please add the according reports from logs to apparmor profiles. I will create a separate ticket to include apparmor in openQA-in-openQA tests.
Updated by okurz 10 months ago
- Copied to action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:M added
Updated by mkittler 10 months ago
This should fix the permission problem: https://github.com/os-autoinst/openQA/pull/5470
Updated by mkittler 10 months ago
- Status changed from Resolved to In Progress
The automatic updating of dependencies also works: https://github.com/os-autoinst/openQA/pull/5469
There's still https://github.com/os-autoinst/openQA/pull/5474 to be merged so I'm keeping the ticket in feedback.
I have also removed cache.txz
and update-cache.sh
from the OBS package repo and the build is still working.
I still have to look at https://build.opensuse.org/package/show/devel:openQA:tested/openQA. So far this package has not been updated yet.
Updated by mkittler 10 months ago
- Status changed from In Progress to Resolved
I merged https://github.com/os-autoinst/openQA/pull/5474 given the "+1" and the approval.
It looks like the package under "tested" was updated without any need for modifications in the scripts: https://build.opensuse.org/package/show/devel:openQA:tested/openQA?expand=1
This is also visible in the logs, e.g. renamed '_service:node_modules:node_modules.obscpio' -> 'node_modules.obscpio'
on http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/lastBuild/consoleFull.
With that we can consider this ticket resolved.
Updated by okurz 10 months ago
- Status changed from Resolved to In Progress
I love the progress that you could make but not quite there yet: https://build.opensuse.org/request/show/1146574 says "Attention, "node_modules.sums" is not mentioned in spec files as source or patch." The problem was actually already mentioned in
http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/lastBuild/consoleFull which also mentions package-lock.json
I will look into that
Updated by okurz 10 months ago
- Due date deleted (
2024-02-16) - Status changed from Feedback to Resolved
https://github.com/os-autoinst/scripts/pull/291 merged. https://build.opensuse.org/request/show/1146602 has all checks passed and is only pending acceptance. And we already had an automatic update of js assets with https://github.com/os-autoinst/openQA/pull/5469 which worked out fine so I guess we are good
Updated by AdamWill 6 months ago · Edited
So I'm finally getting to hitting this in the Fedora package. In Fedora's build system we have no network access ever for any reason, so I cannot use the new approach. Should it be safe to keep using our existing asset cache script - https://src.fedoraproject.org/rpms/openqa/blob/rawhide/f/update-cache.sh - which basically calls tools/generate-packed-assets and then tars up assets/cache and assets/assetpack.db ? Is that still a valid approach with all the NPM and whatever changes? Thanks.
edit: looks like I just needed to make the cache script depend on npm, have it run npm install
before tools/generate-packed-assets
, and include node_modules
in the tarball.