Project

General

Profile

Actions

action #163571

closed

openQA Project (public) - coordination #154768: [saga][epic][ux] State-of-art user experience for openQA

openQA Project (public) - coordination #168127: [epic] Up-to-date Perl stack

cpanspec: Move cronjob for auto updating devel:languages:perl:CPAN-{A..Z} off the obsolete engcloud size:M

Added by tinita 6 months ago. Updated 17 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Start date:
Due date:
2024-12-17
% Done:

0%

Estimated time:

Description

Motivation

engcloud will be switched off, so we need to move the process to a different cloud server.
See also https://confluence.suse.com/pages/viewpage.action?pageId=450594448
https://build.opensuse.org/project/show/devel:languages:perl:CPAN-A etc. are regularly updated with the newest CPAN uploads.

Acceptance Criteria

AC1: Everything is moved from engcloud to a different provider

Suggestions

  • The daily cronjob needs to be moved.
  • The cache files under ~/obs-mirror/obs-cache/ need to be copied. They contain the information about all modules and versions. Without the cache it can take several days to query the API to gather the information again
  • Consider github actions and using git LFS for storing the state for each lettered project and the list of packages where cpanspec fails
    • when working in a github repository with GHA the state could be commited with git LFS into a separate branch with even just amending the last commit instead of adding new commits
  • Consider OpenPlatform (Harvester) as a fallback solution if GitHub does not work out: https://op-prg2.openplatform.suse.com/dashboard/home
  • We also have munin running on that server with useful information about the build status, e.g. http://10.86.1.204/munin/obs-cpan-day.html . Ideally this should also be moved, but it doesn't have to run on the same server. The code for the munin data is also in https://github.com/openSUSE/autoupdate-perl

Related issues 3 (1 open2 closed)

Copied from QA (public) - action #160688: cpanspec: Run autoupdate for devel:languages:perl in github workflow size:MResolvedtinita2024-05-212024-07-13

Actions
Copied to QA (public) - action #169633: Setup an environment to run cpan cronjobs size:SResolvedtinita2024-11-26

Actions
Copied to QA (public) - action #174709: Select better network for obs-cpan-mirror VM on op-prg2 harvesterNew

Actions
Actions #1

Updated by tinita 6 months ago

  • Copied from action #160688: cpanspec: Run autoupdate for devel:languages:perl in github workflow size:M added
Actions #2

Updated by okurz 6 months ago

  • Target version set to Ready
Actions #3

Updated by okurz 6 months ago

@tinita I would like to reconsider this with you. As this is about updating packages on OBS we should try really hard to use a platform that is available to the community, i.e. not SUSE internal openplatform. I am sure we can find a good solution.

Actions #4

Updated by okurz 6 months ago

  • Subject changed from cpanspec: Move cronjob for auto updating devel:languages:perl:CPAN-{A..Z} to openplatform to cpanspec: Move cronjob for auto updating devel:languages:perl:CPAN-{A..Z} off the obsolete engcloud
  • Description updated (diff)
Actions #5

Updated by okurz 6 months ago

  • Description updated (diff)
Actions #6

Updated by livdywan 5 months ago

  • Subject changed from cpanspec: Move cronjob for auto updating devel:languages:perl:CPAN-{A..Z} off the obsolete engcloud to cpanspec: Move cronjob for auto updating devel:languages:perl:CPAN-{A..Z} off the obsolete engcloud size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #7

Updated by tinita 5 months ago

I asked in Slack about when engcloud will be shutdown.

Actions #8

Updated by tinita 4 months ago

  • Priority changed from Normal to High

The plan is to shutdown "in September", so I guess this should be high prio now.

Actions #9

Updated by tinita 4 months ago

We don't have a backup for the obs mirror host, so I started to rsync files, in case the machine gets turned off.

Actions #10

Updated by livdywan 4 months ago

tinita wrote in #note-9:

We don't have a backup for the obs mirror host, so I started to rsync files, in case the machine gets turned off.

Where do the files end up? Somewhere on osd?

Actions #11

Updated by tinita 4 months ago

Where do the files end up? Somewhere on osd?

on my own laptop for now

Actions #12

Updated by livdywan 3 months ago

We should discuss this again. As this is a High ticket, but effectively currently treated as a Low ticket. So either we figure out how to help out here to get it done asap or really re-consider the priority of it.

Actions #13

Updated by livdywan 3 months ago

Will try and bring it up at the next opportunity this week

Actions #14

Updated by tinita 3 months ago

  • Description updated (diff)
Actions #15

Updated by livdywan 3 months ago

We looked into that last week and created virtual machines via Virtualization Management in the qe namespace to test it out. The machine could also be setup by someone else to help @tinita. This includes checking what image and resources to use and how to make the machine available to the outside and ensuring multiple people in the team have access to it.

Actions #16

Updated by tinita 3 months ago

The cloud server was shutdown in the meantime. This should be done soon.

Actions #17

Updated by okurz 3 months ago

  • Priority changed from High to Normal

As we discussed the cost of delay is that due to the absence of the cache the update procedure becomes lengthy and eventually packages or newer versions will be missing from devel:languages:perl which can become annoying for manual submitters.

Actions #18

Updated by okurz 3 months ago

  • Parent task set to #168127
Actions #19

Updated by livdywan about 2 months ago

  • Status changed from Workable to Blocked

livdywan wrote in #note-15:

We looked into that last week and created virtual machines via Virtualization Management in the qe namespace to test it out. The machine could also be setup by someone else to help @tinita. This includes checking what image and resources to use and how to make the machine available to the outside and ensuring multiple people in the team have access to it.

I think at this point we need a new ticket to block on since I have the feeling people forget we need a new server here before we can continue. And moving this back to High likely won't help us.

Actions #20

Updated by livdywan about 2 months ago

  • Copied to action #169633: Setup an environment to run cpan cronjobs size:S added
Actions #21

Updated by tinita about 1 month ago

  • Status changed from Blocked to In Progress

#169633 done

Actions #22

Updated by openqa_review about 1 month ago

  • Due date set to 2024-11-29

Setting due date based on mean cycle time of SUSE QE Tools

Actions #23

Updated by okurz 29 days ago

  • Status changed from In Progress to Workable
Actions #24

Updated by livdywan 28 days ago

  • Due date deleted (2024-11-29)
Actions #25

Updated by tinita 24 days ago

The cluster I created the VM on is in maintenance. It is supposed to take at least until the end of tuesday (tomorrow), so I can't work on it.

Actions #26

Updated by tinita 24 days ago

  • Status changed from Workable to In Progress

Actually, I found the IP in my shell history and just tried to connect. The VM is still there, up and running, just the cluster in openplatform is gone 🤷

I now ran several parts of the updating jobs and need to fix a few things, but it looks good.
Since the cluster is under maintenance, I won't schedule any cronjob for now.

Actions #27

Updated by openqa_review 24 days ago

  • Due date set to 2024-12-17

Setting due date based on mean cycle time of SUSE QE Tools

Actions #28

Updated by tinita 22 days ago · Edited

Installed munin for some basic monitoring: http://10.144.110.163/munin/

edit: new ip

Actions #29

Updated by tinita 22 days ago

Configured logrotate.

https://github.com/openSUSE/autoupdate-perl/pull/10

Updated a bunch of packages. Will continue tomorrow.

Taking regular backups is not yet supported in the currently used harvester version (1.2).
One can take manual VM Snapshots, VM Backups and Volume Snapshots. I tried out all three. The VM Backup has a note "The current backup target does not match the existing one." Not sure how to influence that as I couldn't select anything when creating it.

Still missing: I would like to have a basic alerting.

Actions #30

Updated by tinita 22 days ago

Actions #32

Updated by tinita 21 days ago

All packages are uptodate.
I enabled the cronjobs:

IMAGE=registry.opensuse.org/home/tinita/cpanspec/container/containers/opensuse/autoupdate-perl
CPANSPEC_WRAP_BUILD_COMMAND="chgrp -R cpan ..; chmod -R g+rx ..; chmod -R g+rwx .; su cpanbuild -c '\$COMMAND'"

5 3 * * * podman run -i --rm -e HOME=/home/$USER -v$HOME:/home/$USER $IMAGE /home/$USER/autoupdate-perl/bin/fetch-cpan.sh >> $HOME/obs-mirror/logs/cpan.log 2>&1
5 4 * * * podman run -i --rm -e HOME=/home/$USER -v$HOME:/home/$USER -e CPANSPEC_WRAP_BUILD_COMMAND $IMAGE /home/$USER/autoupdate-perl/bin/update-dlp-cpan.sh >> $HOME/obs-mirror/logs/cpanupdate.log 2>&1

The vlan had to be changed, and the IP address is fixed now: 10.144.110.163

Actions #33

Updated by okurz 21 days ago

https://gitlab.suse.de/suse/wiki/-/merge_requests/6 is merged. You might want to consider https://github.com/openSUSE/os-update/ and maybe also systemd-mail-status, also mentioned on the os-update wiki page. Both are in openSUSE so just zypper in os-update systemd-mail-status and configure

Actions #34

Updated by tinita 17 days ago

  • Status changed from In Progress to Feedback

I added hopefully everything necessary to setup the server here: https://confluence.suse.com/pages/viewpage.action?pageId=450594448

Actions #35

Updated by tinita 17 days ago

  • Status changed from Feedback to Resolved
Actions #36

Updated by okurz 3 days ago

  • Copied to action #174709: Select better network for obs-cpan-mirror VM on op-prg2 harvester added
Actions

Also available in: Atom PDF