Project

General

Profile

Actions

action #153427

closed

coordination #154768: [saga][epic][ux] State-of-art user experience for openQA

coordination #157510: [epic] Up-to-date JavaScript stack

Improve updating cached assets size:M

Added by tinita 11 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-01-05
Due date:
% Done:

0%

Estimated time:

Description

Motivation

In #153145 we saw that the dist/rpm/update-cache.sh script does not work in the case when we add new assets or change urls for existing assets.
If you make a pull request with a new url, then it will fail in OBS because it has the old cache.
But running update-cache.sh will use the upstream openQA git repo which has the old url.
So it's a chicken-egg problem.

That also is related to when others try to build openQA outside the context of OBS, see recent discussions in https://app.element.io/#/room/#openqa:opensuse.org

Acceptance criteria

  • AC1: It is not necessary to manually update the cached assets for rpm packaging when a remotly fetched asset is changed (in assetpack.def)

Suggestions


Related issues 2 (0 open2 closed)

Copied from openQA Project (public) - action #153145: [ci] Failing openQA pull requests in generate-packed-assets stepResolvedtinita2024-01-052024-01-20

Actions
Copied to openQA Project (public) - action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:MResolvedjbaier_cz

Actions
Actions #1

Updated by tinita 11 months ago

  • Copied from action #153145: [ci] Failing openQA pull requests in generate-packed-assets step added
Actions #2

Updated by okurz 11 months ago

  • Due date deleted (2024-01-20)
  • Target version set to Ready
Actions #3

Updated by okurz 11 months ago

  • Tags set to reactive work
Actions #4

Updated by okurz 11 months ago

  • Subject changed from Improve updating cached assets to Improve updating cached assets size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by mkittler 10 months ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions #6

Updated by mkittler 10 months ago · Edited

It is true that one has generally internet access on OBS service runs. However, since we cannot use any existing services without extra effort and our current way of fetching assets is a bit involved this is not so easy to exploit.

There are multiple ways to approach this and none of them seem very straight forward:

  1. We depend entirely on npm modules. Then we could probably use https://github.com/openSUSE/obs-service-node_modules like cockpit.
    1. Switching to webpack would probably help but doesn't solve all of it because likely not all of our dependencies are packaged as npm modules. In addition, when I remember correctly, our assetpack scripts did things that were not easily replicable with webpack when I tried so this would be a huge refactoring.
  2. We write our own obs-service. It would just invoke our existing script for updating the cache.
    1. I'm not sure about the deployment. Probably we needed to create an rpm package (like the ones found on https://build.opensuse.org/project/show/openSUSE:Tools) and ask OBS admins to deploy it.
    2. The problem might be that our script depends on Mojolicious and various other Perl packages. In fact, we'd probably require almost all of openQA's normal dependencies as we basically just start the web application. Probably deploying an obs-service package which such a huge dependency trail is not acceptable for OBS admins and maybe not all dependencies are even resolvable on whatever system they're using. (If this was simple enough than @coolo would probably have already implemented it this way back in the day.)
  3. Use an existing service which would probably mean parsing assetpack.def and generating a list of files to be downloaded from that in the format expected by an existing service:
    1. https://github.com/openSUSE/obs-service-download_files reads the URLs from the specfile itself which is probably not what we want.
    2. https://github.com/openSUSE/obs-service-download_url lacks a README so I we had to reverse engineer for details on it.
    3. We might not be in control of the directory structure where the downloaded files are put under but we need that as we need to download files where assetpack would expect them.
      1. We could of course move files around at build time (in the rpm %build section) but that would not be very nice.
    4. We have tons of individual files to be downloaded. It would probably not be nice to clutter OBS with that. Our current asset update script solves that by simply creating the archive cache.txz instead of adding the files to OBS individually. Not sure whether that would be possible using such a service.
  4. We write our own obs-service. It would not invoke our existing script but be much simpler instead.
    1. It would only parse assetpack.def and download files where assetpack would expect them.
    2. To avoid cluttering OBS we could create the archive cache.txz as before.

I asked on the OBS channel what's the best approach is. Because IRC is not very persistent:

[14:37] Any idea what the best approach for caching external assets for openQA builds on OBS would be?
[14:38] I suppose it would boil down to making use on an OBS service of some kind. I already created a list of some options: https://progress.opensuse.org/issues/153427#note-6
[14:38] Unfortunately none of those options is particularly easy.
[14:41] I'm aware that other complex packages rely on the maintainer updating the package to run some script locally (like the Grafana package that comes with its own Makefile which in turn invokes yarn and tons of things I don't even want to know). That's what we did so far for openQA as well but it would be desirable to automate this as we generally build new packages as there are new changes in Git and we even have OBS
[14:41] package builds as CI checks.

I'm waiting for feedback. I was suggested to read https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.scm_bridge as it might not make sense anymore putting so much effort into the old handling of sources on OBS.

Actions #7

Updated by mkittler 10 months ago

  • Status changed from In Progress to Feedback
Actions #8

Updated by mkittler 10 months ago

[15:56] in short, obs is just following the sources of a git server and is not doing any kind of modifications anymore
[15:56] so entire workflow needs to be eg. in gitea actions or external services. And entire source state of entire project can be verified via a single identifier
[15:57] there is also a remoteasset implementation, but most likely not helpful for you

That would mean we'd had some other job (e.g. a GitHub action) that pushes the assets we need on some Git branch that is then consumed by OBS (like it currently does for our normal source code or with the SCM bridge approach when we migrate to that). That in turn means:

  1. We need to run a CI action on every PR which is potentially pushing changes on a newly created Git branch that is then configured to be used by the packaging CI checks.
    1. "potentially" because the action would likely only be triggered (or do anything) if assetpack.def was changed.
    2. The CI checks for packaging needed to wait until the action pushing changes has completed. This might not be easy to solve because the CI checks for packaging are simply triggered by a webhook. So we cannot make use of e.g. the dependency handling within CircleCI.
  2. We need to run a CI action after every merged PR which is potentially pushing changes on a dedicated Git branch that is configured to be used by our packaging on devel:openQA.
    1. Since the Git branch the changes would be pushed on is just one for the whole repository we need to synchronize the CI actions (so there's no conflict when PRs are merged shortly after another).

That's definitely some work, or am I over-complicating things?


As a start we could just create an "assets" Git branch manually and configure our packaging to use it.

Actions #9

Updated by okurz 10 months ago

That's definitely some work, or am I over-complicating things?

I think you are not overcomplicating things if we really want to build and package openQA and the assets within OBS.

I would prefer to not maintain more branches. We already have master and gh-pages. Admittedly gh-pages is really low effort but still. How about we consider for a chance how one would build an openQA container from the git sources. How would that work? And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?

Actions #10

Updated by mkittler 10 months ago · Edited

How about we consider for a chance how one would build an openQA container from the git sources. How would that work?

Regardless of using a container or not, one would have to download assets on the first launch of the web service - unless we also improve that. Having all files that would be downloaded directly in our on Git repo at the right place would of course improve that.

And considering that how about building the assets archive by GHA on the master branch whenever necessary and creating pull requests to master to include the archive directly there?

I would not create an archive then but simply add the files to Git where they would end up anyway. Otherwise we would add a binary file in Git and the user would have to extract the archive first (like the rpm build currently does). I personally don't have a problem with having those external files in our Git repo in general although we need to clarify the licensing¹.

Note that asset pack produces a different output for different modes (development/testing/ production). We also actually use this to have minimized files in production and to disable certain effects when testing (via Selenium). I suppose this is not going to be a problem as long as we really only check-in the files that would be downloaded. The build script could then invoke the web application once to generate the remaining files (so the final package is complete and doesn't rely on computing files anymore at startup which would probably also run into permission errors).


¹In a personal repository I actually did the same and just checked in the licensing files that came by installing the npm package anyway (e.g. https://github.com/Martchus/arch-repo-manager/tree/master/srv/static/node_modules/xterm). In our case we don't use npm but could just download and add the LICENSING files manually.

Actions #11

Updated by mkittler 10 months ago

RFC-PR for adding assets directly to our Git repository: https://github.com/os-autoinst/openQA/pull/5454

Actions #12

Updated by mkittler 10 months ago · Edited

RFC-PR for providing assets via npm: https://github.com/os-autoinst/openQA/pull/5463

Unfortunately this doesn't seem to cut it. The obs-service-node_modules service can apparently not be used that way. I only realize now that cockpit also only uses it with mode="manual" which means it doesn't actually run on OBS.
Besides that, it would also require more additional work to adapt the CI. I already put some effort into it but there are still problems.

Maybe we should reiterate the options we have (e.g. in the next unblock meeting).

Actions #13

Updated by okurz 10 months ago

  • Due date set to 2024-02-16
Actions #14

Updated by mkittler 10 months ago · Edited

It looks like the problem was fixable after all and the service runs now on my PR branch on OBS: https://build.opensuse.org/package/show/devel:openQA:GitHub:os-autoinst:openQA:PR-5463/openQA
The builds is still running as I write this comment but it looks promising. I'll download the rpm and test whether it actually works locally.

I was also able to adapt CI checks. There are still some fixes remaining as now the style check wants to go over files from the node_modules directory.

I also still need to figure out that handling of fonts in production mode (where the URLs are not correctly resolved).

Actions #15

Updated by mkittler 10 months ago

  • After my latest changes CI checks should work.
  • I tested the RPMs locally and found issues. It should be fixed after my latest changes but I still need to check that once the RPMs have been rebuilt.
  • The mentioned issue about resolving URLs could be fixed by amending the Fetch-pipeline of AssetPack.
  • I linked the local-npm-registry package in our devel repos to get Leap builds going. I created https://build.opensuse.org/request/show/1146197 for SLE.
  • I checked https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit used by our Tumbleweed submissions. It looks like the script will cope with the newly added service without modifications. (I haven't tested it yet. Maybe we need slight modifications after all but this shouldn't be a big deal.)
Actions #16

Updated by okurz 10 months ago

Sounds like a plan! I think we are at the point where after you fix remaining CI failures we can merge your PR which goes way beyond what I officially thought we could come up with improvements.

Actions #17

Updated by mkittler 10 months ago

  • Status changed from Feedback to In Progress
Actions #19

Updated by okurz 10 months ago

https://github.com/os-autoinst/openQA/pull/5463 merged and apparently already deployed to o3 as it caused a problem

https://suse.slack.com/archives/C02CANHLANP/p1707834084412189?thread_ts=1707834084.412189&cid=C02CANHLANP
Reports

 openqa-gru[7430]: Can't open file "/usr/share/openqa/script/../assets/../node_modules/datatables.net-bs4/css/dataTables.bootstrap4.min.css": Permission denied

Mitigated by fvogt

I did sudo aa-complain /usr/share/openqa/script/openqa and it works again.

Please add the according reports from logs to apparmor profiles. I will create a separate ticket to include apparmor in openQA-in-openQA tests.

Actions #20

Updated by okurz 10 months ago

  • Copied to action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:M added
Actions #21

Updated by mkittler 10 months ago

This should fix the permission problem: https://github.com/os-autoinst/openQA/pull/5470

Actions #22

Updated by favogt 10 months ago

  • Status changed from In Progress to Resolved

Looks like o3 is up and running with the AppArmor profile in enforcing mode again.

Actions #23

Updated by mkittler 10 months ago

  • Status changed from Resolved to In Progress

The automatic updating of dependencies also works: https://github.com/os-autoinst/openQA/pull/5469

There's still https://github.com/os-autoinst/openQA/pull/5474 to be merged so I'm keeping the ticket in feedback.

I have also removed cache.txz and update-cache.sh from the OBS package repo and the build is still working.

I still have to look at https://build.opensuse.org/package/show/devel:openQA:tested/openQA. So far this package has not been updated yet.

Actions #24

Updated by mkittler 10 months ago

  • Status changed from In Progress to Resolved

I merged https://github.com/os-autoinst/openQA/pull/5474 given the "+1" and the approval.

It looks like the package under "tested" was updated without any need for modifications in the scripts: https://build.opensuse.org/package/show/devel:openQA:tested/openQA?expand=1
This is also visible in the logs, e.g. renamed '_service:node_modules:node_modules.obscpio' -> 'node_modules.obscpio' on http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/lastBuild/consoleFull.

With that we can consider this ticket resolved.

Actions #25

Updated by okurz 10 months ago

  • Status changed from Resolved to In Progress

I love the progress that you could make but not quite there yet: https://build.opensuse.org/request/show/1146574 says "Attention, "node_modules.sums" is not mentioned in spec files as source or patch." The problem was actually already mentioned in
http://jenkins.qa.suse.de/job/submit-openQA-TW-to-oS_Fctry/lastBuild/consoleFull which also mentions package-lock.json

I will look into that

Actions #26

Updated by okurz 10 months ago

  • Status changed from In Progress to Feedback
Actions #27

Updated by okurz 10 months ago

  • Due date deleted (2024-02-16)
  • Status changed from Feedback to Resolved

https://github.com/os-autoinst/scripts/pull/291 merged. https://build.opensuse.org/request/show/1146602 has all checks passed and is only pending acceptance. And we already had an automatic update of js assets with https://github.com/os-autoinst/openQA/pull/5469 which worked out fine so I guess we are good

Actions #28

Updated by okurz 9 months ago

  • Parent task set to #157510
Actions #29

Updated by AdamWill 6 months ago · Edited

So I'm finally getting to hitting this in the Fedora package. In Fedora's build system we have no network access ever for any reason, so I cannot use the new approach. Should it be safe to keep using our existing asset cache script - https://src.fedoraproject.org/rpms/openqa/blob/rawhide/f/update-cache.sh - which basically calls tools/generate-packed-assets and then tars up assets/cache and assets/assetpack.db ? Is that still a valid approach with all the NPM and whatever changes? Thanks.

edit: looks like I just needed to make the cache script depend on npm, have it run npm install before tools/generate-packed-assets, and include node_modules in the tarball.

Actions

Also available in: Atom PDF