Project

General

Profile

Actions
Copy link

action #150845

closed

openqaworker-arm22 broken due to packages automatically removed size:M

Added by okurz 6 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-11-14
Due date:
2023-11-29
% Done:

0%

Estimated time:
Tags:
arm, o3, infra

Description

Observation

https://matrix.to/#/!dRljORKAiNJcGEDbYA:opensuse.org/$NAjNVuJkCp9sUvGegwMaVkh6Saenhly0cX51EM8BZJc?via=matrix.org&via=opensuse.org&via=im.f3l.de

Hi! It seems openqaworker-arm22 worker is broken: https://openqa.opensuse.org/tests/3719839
qemu-img seems to be missing! Any idea why ? and who could access it to fix it? Looks like it is not reachable from Ariel??

Acceptance criteria

  • AC1: openQA jobs on openqaworker-arm22 are ok again
  • AC2: We know why packages had been uninstalled

Suggestions

  • Stop the worker instances
  • Investigate the problem regarding packages removed causing qemu missing
  • Fix the problem, probably in openqa-auto-update or openqa-continuous-update, potentially regression from latest change to do explicit zypper ref
  • Ensure the problem is fixed for good on also openqaworker-arm22
  • Ensure openqaworker-arm22 is fully up-to-date and automatic update services are enabled
  • Ensure openQA jobs on openqaworker-arm22 are ok again

Related issues 2 (0 open — 2 closed)

Related to openQA Infrastructure - action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:MResolvedtinita2023-11-15

Actions
Related to openQA Infrastructure - action #150920: openqaworker-arm22 is unable to join download.opensuse.org in parallel tests = tap mode size:MResolvednicksinger2023-11-15

Actions
Actions
Copy link
 #1

Updated by okurz 6 months ago

  • Priority changed from Urgent to Immediate

On request by guillaume_g as the worker is messing up tests making this "immediate"

Actions
Copy link
 #2

Updated by mkittler 6 months ago

  • Status changed from New to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #3

Updated by mkittler 6 months ago

I cannot access via SSH and proxy-opensuse.suse.de as I am being asked for a password. I cannot access via the public proxy which is known to be broken. I can also not access via IPMI because the jump host seems down (ssh: Could not resolve hostname oqa-jumpy.dmz-prg2.suse.org: Temporary failure in name resolution). I cannot even reach the o3 web UI via https://openqa.opensuse.org. So I currently see no way going forward with this ticket. I'll try figure out whether there's currently some more severe problem.

Actions
Copy link
 #4

Updated by mkittler 6 months ago

I now got access via IPMI. It looks like OpenSSH is not installed on that machine. QEMU was also not installed so that is why the backend fails to start. That's rather strange. Maybe some continuous/automatic update accidentally uninstalled packages.

Actions
Copy link
 #5

Updated by mkittler 6 months ago

According to /var/log/zypper.log-20231111.xz many packages have been uninstalled on that day. There were apparently issues with getting repositories:

2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1200 URL: http://download.opensuse.org/repositories/devel:openQA/15.5/repodata/repomd.xml
2023-11-10 16:04:51 <3> openqaworker-arm22(37449) [zypp::media] MediaCurl.cc(doGetFileCopyFile):1265 curl error: 7: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server, temp file size 0 bytes.
2023-11-10 16:04:51 <5> openqaworker-arm22(37449) [zypp-core] Exception.cc(log):186 MediaCurl.cc(evaluateCurlCode):960 THROW:    Download (curl) error for 'http://download.opensuse.org/repositories/devel:openQA/15.5/repodata/repomd.xml':
2023-11-10 16:04:51 <5> openqaworker-arm22(37449) [zypp-core] Exception.cc(log):186 Error code: Connection failed
2023-11-10 16:04:51 <5> openqaworker-arm22(37449) [zypp-core] Exception.cc(log):186 Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server

Followed by a huge list of packages being uninstalled:

2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypp::solver] SATResolver.cc(solving):751 ....Solver end
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypp::solver] SATResolver.cc(resolvePool):921 SATResolver::resolvePool() done. Ret:1
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper] solve-commit.cc(solve_and_commit):677 got solution, showing summary
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper] Summary.cc(readPool):130 Pool contains 1574 items.
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):131 Install summary:
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(8)Mesa-libva-22.3.5-150500.75.2.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(10)aaa_base-extras-84.87+git20180409.04c9dae-150300.10.6.2.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(11)acl-2.2.52-4.3.1.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(12)adjtimex-1.29-1.26.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(17)alsa-oss-1.1.8-150300.12.3.2.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(18)alsa-plugins-1.2.7.1-150500.1.1.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(19)alsa-plugins-speexrate-1.2.7.1-150500.1.1.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(20)alsa-plugins-upmix-1.2.7.1-150500.1.1.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(30)at-spi2-atk-common-2.38.0-150400.1.9.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(31)at-spi2-atk-gtk2-2.38.0-150400.1.9.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(35)attr-2.4.47-2.19.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_o(36)audit-3.0.6-150400.4.13.1.aarch64(@System)
…
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_u(1570)pattern:base-20200505-lp155.10.5.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_u(1571)pattern:documentation-20200505-lp155.10.5.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_(1572)pattern:enhanced_base-20200505-lp155.10.5.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_u(1573)pattern:minimal_base-20200505-lp155.10.5.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_u(1574)pattern:sw_management-20200505-lp155.10.5.aarch64(@System)
2023-11-10 16:04:51 <1> openqaworker-arm22(37449) [zypper++] Summary.cc(readPool):171 <uninstall> I_Ts_(1575)pattern:yast2_basis-20220411-lp155.2.4.aarch64(@System)

A human would have never gone along with uninstalling all these packages so I'm pretty sure it was the continuous update.

I uploaded the full log to https://w3.nue.suse.com/~mkittler/arm22-zypper.log-20231111.xz.

Actions
Copy link
 #6

Updated by mkittler 6 months ago

I extracted the list of uninstalled packages from the logs and installed them again via:

zypper in Mesa-libva aaa_base-extras acl adjtimex alsa-oss alsa-plugins alsa-plugins-speexrate alsa-plugins-upmix at-spi2-atk-common at-spi2-atk-gtk2 attr audit autofs bind-utils blog breezy busybox-static checkmedia cifs-utils cpio-mt crda cron cronie cups cups-client cups-filters cyrus-sasl cyrus-sasl-crammd5 cyrus-sasl-digestmd5 cyrus-sasl-gssapi cyrus-sasl-plain dbus-1-glib dbus-1-glib-tool dbus-1-x11 dconf diffstat dirmngr dmraid dos2unix dracut ed ffmpeg-4 fuse fuse3 gcr-ssh-agent gcr-viewer gdk-pixbuf-thumbnailer geoclue2 ghostscript ghostscript-x11 git-gui gitk glibc-extra gpart gpm groff groff-full gsettings-backend-dconf gstreamer-libnice gstreamer-plugins-good-gtk gtk2-immodule-amharic gtk2-immodule-inuktitut gtk2-immodule-thai gtk2-immodule-tigrigna gtk2-immodule-vietnamese gtk2-tools gtk3-immodule-amharic gtk3-immodule-inuktitut gtk3-immodule-thai gtk3-immodule-tigrigna gtk3-immodule-vietnamese gtk3-tools gvfs-backend-samba gvfs-backends gvfs-fuse gxditview haveged htop ifplugd iputils irqbalance irqbalance-ui joe kernel-default kernel-default-extra kernel-default-optional kexec-tools keyutils klogd kpartx libSDL2-2_0-0 libXdmcp6 libXfont2-2 libXvnc1 libauparse0 libavahi-glib1 libavcodec58_134 libavdevice58_13 libavfilter7_110 libavformat58_76 libavresample4_0 libavutil56_70 libbd_btrfs2 libblogger2 libbluray2 libbrotlienc1 libbsd0 libcacard0 libcap-progs libcapstone4 libcdio16 libcdio19 libcdio_cdda2 libcdio_paranoia2 libcelt0-2 libcetcd0 libcupscgi1 libcupsimage2 libcupsmime1 libcupsppdc1 libdaemon0 libdconf1 libdpdk-20_0 libefa1 libevent-2_1-8 libfdt1 libfmt8 libfontenc1 libfstrm0 libfuse3-3 libgpm2 libhavege2 libibverbs libibverbs1 libimagequant0 libinput-udev libiw30 libjack0 libjasper4 libldb2 liblttng-ust0 libmaxminddb0 libmediacheck6 libmlx4-1 libmlx5-1 libmm-glib0 libmpath0 libmtp-udev libmtp9 libnetpbm11 libnfs8 libnl3-200 libnotify4 libnscd1 libnss_usrfiles2 libntfs-3g87 libnuma1 libopenvswitch-2_14-0 libpcap1 libpipeline1 libpoppler-cpp0 libpostproc55_9 libprotobuf-c1 libqpdf26 libqt5-qtbase-platformtheme-gtk3 libqt5-qtstyleplugins-platformtheme-gtk2 libqt5-qttranslations librados2 librbd1 librdmacm1 libsamba-policy0-python3 libscg1_0 libschily1_0 libsensors4 libsgutils2-1_47-2 libslirp0 libsmi libsmi2 libsmputils1-1 libsnappy1 libsnmp40 libsodium23 libspice-server1 libswresample3_9 libswscale5_9 libtalloc2 libtdb1 libtevent0 libtss2-esys0 libtss2-mu0 libtss2-rc0 libtss2-sys1 libudisks2-0_btrfs libunbound2 liburcu6 liburing1 liburing2 libusbredirparser1 libuv1 libva-glx2 libva-x11-2 libvdeplug3 libvdpau1 libvirglrenderer1 libvmaf1 libwrap0 libxkbfile1 libyajl2 libzimg2 luit mailx man metamail mkfontdir mkfontscale mkisofs mokutil mpg123-openal mpt-status multipath-tools net-snmp netcat-openbsd netpbm nfs-client nfsidmap nscd ntfs-3g ntfsprogs numactl openSUSE-signkey-cert openal-soft openldap2-client openssh openssh-askpass openssh-askpass-gnome openssh-server openvswitch os-autoinst-openvswitch os-prober ovmf pam_pwquality patterns-base-apparmor patterns-base-base patterns-base-documentation patterns-base-enhanced_base patterns-base-minimal_base patterns-base-sw_management patterns-yast-yast2_basis perl-Class-XSAccessor perl-Devel-Caller perl-Devel-LexAlias perl-Inline-Python perl-JSON-XS perl-PadWalker perl-Readonly-XS perl-SNMP perl-Sub-Name perl-Term-ReadKey perl-Term-ReadLine-Gnu perl-core-DB_File pngquant poppler-tools postfix prctl procinfo procmail psutils python3-Pillow python3-Pillow-tk python3-brotlipy python3-dbm python3-dulwich python3-gevent python3-geventhttpclient python3-gobject-cairo python3-gpg python3-greenlet python3-ldb python3-patiencediff python3-pycairo python3-talloc python3-tdb python3-tevent python3-tk qemu qemu-arm qemu-audio-spice qemu-block-curl qemu-block-nfs qemu-block-rbd qemu-chardev-spice qemu-hw-display-qxl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga qemu-hw-usb-host qemu-hw-usb-redirect qemu-hw-usb-smartcard qemu-ksm qemu-linux-user qemu-tools qemu-ui-curses qemu-ui-gtk qemu-ui-opengl qemu-ui-spice-app qemu-ui-spice-core rdma-core rdma-ndd rpcbind samba samba-client samba-client-libs samba-libs samba-libs-python3 samba-python3 screen sensors setserial setxkbmap sg3_utils shim site-config smartmontools smp_utils snmp-mibs spax sqlite3-devel star star-rmt strace sudo-plugin-python sysstat system-group-audit system-user-srvGeoClue systemd-coredump systemd-doc systemd-sysvinit tar-rmt tcpdump terminfo terminfo-iterm terminfo-screen time tnftp traceroute udev-configure-printer unbound-anchor unzip-doc usbutils vim vlan wicked wireless-tools x3270 xauth xclock xdotool xinit xkbcomp xmodmap xorg-x11-Xvnc xorg-x11-Xvnc-module xrdb xsetroot xterm yast2-alternatives yast2-snapper yast2-vm zisofs-tools zstd pattern:apparmor pattern:base pattern:documentation pattern:enhanced_base pattern:minimal_base pattern:sw_management pattern:yast2_basis

This worked but I also got the following errors along the way:

Problem retrieving files from 'devel_openQA'.
Timeout exceeded when accessing 'http://download.opensuse.org/repositories/devel:openQA/15.5/repodata/repomd.xml'.
Please see the above error message for a hint.
Warning: Skipping repository 'devel_openQA' because of the above error.
Retrieving repository 'devel_openQA_Leap' metadata ..................................................................................................................................................................................................................................................................[error]
Repository 'devel_openQA_Leap' is invalid.
[devel_openQA_Leap|http://download.opensuse.org/repositories/devel:openQA:Leap:15.5/15.5] Valid metadata not found at specified URL
History:
 - [devel_openQA_Leap|http://download.opensuse.org/repositories/devel:openQA:Leap:15.5/15.5] Repository type can't be determined.

Please check if the URIs defined for this repository are pointing to a valid repository.
Warning: Skipping repository 'devel_openQA_Leap' because of the above error.
Some of the repositories have not been refreshed because of an error.
…
Retrieving: libbrotlienc1-1.0.7-3.3.1.aarch64.rpm ...................................................................................................................................................................................................................................................................[error]
Download (curl) error for 'http://download.opensuse.org/distribution/leap/15.5/repo/oss/aarch64/libbrotlienc1-1.0.7-3.3.1.aarch64.rpm':
Error code: HTTP response: 500
Error message: The requested URL returned error: 500

Abort, retry, ignore? [a/r/i/...? shows all options] (a): r
Retrieving: libbrotlienc1-1.0.7-3.3.1.aarch64.rpm ....................................................................................................................................................................................................................................................................[done]

Maybe our devel repo was just being published right now. That can happen and shouldn't lead to a huge uninstall of other packages. That the aarch64 repository returned a 500 error is on the other hand rather worrisome. There's likely a problem in the mirror infrastructure. Maybe something to ask @andriinikitin about.

Actions
Copy link
 #7

Updated by mkittler 6 months ago

Strangely, the log of openqa-continuous-update does not contain any hint that qmeu, openssh or other packages from that huge list would be uninstalled by it. The log starts at Sep 08 so that would definitely be far enough. It also does contain mentionings of conflicts but those also show that the service would not proceed in such a case:

Nov 11 12:25:31 openqaworker-arm22 systemd[1]: Stopped Continuously deploys openQA, see https://progress.opensuse.org/issues/105379.
Nov 11 12:25:31 openqaworker-arm22 systemd[1]: Started Continuously deploys openQA, see https://progress.opensuse.org/issues/105379.
Nov 11 12:25:31 openqaworker-arm22 openqa-continuous-update[13121]: devel:openQA looks good for Leap 15.5 (aarch64)
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'devel_openQA' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'devel_openQA_Leap' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'openSUSE-Leap-15.5-1' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Update repository of openSUSE Backports' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Non-OSS Repository' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Open H.264 Codec (openSUSE Leap)' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Main Repository' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Update repository with updates from SUSE Linux Enterprise 15' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Main Update Repository' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: Repository 'Update Repository (Non-Oss)' is up to date.
Nov 11 12:25:32 openqaworker-arm22 openqa-continuous-update[13132]: All repositories have been refreshed.
Nov 11 12:25:33 openqaworker-arm22 openqa-continuous-update[13167]: Loading repository data...
Nov 11 12:25:33 openqaworker-arm22 openqa-continuous-update[13167]: Reading installed packages...
Nov 11 12:25:34 openqaworker-arm22 openqa-continuous-update[13167]: Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Nov 11 12:25:34 openqaworker-arm22 openqa-continuous-update[13167]: Computing distribution upgrade...
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: 4 Problems:
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Problem: problem with the installed perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Problem: problem with the installed perl-Selenium-Remote-Driver-1.49-lp155.2.1.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Problem: problem with the installed tesseract-ocr-5.3.1-lp155.38.8.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Problem: problem with the installed libtesseract5-5.3.1-lp155.38.8.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Problem: problem with the installed perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]:  Solution 1: install perl-Cpanel-JSON-XS-4.36-bp155.2.3.1.aarch64 from vendor openSUSE
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]:   replacing perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64 from vendor obs://build.opensuse.org/devel:openQA
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]:  Solution 2: keep obsolete perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64
Nov 11 12:25:35 openqaworker-arm22 openqa-continuous-update[13167]: Choose from above solutions by number or skip, retry or cancel [1/2/s/r/c/d/?] (c): c
Nov 11 12:25:35 openqaworker-arm22 systemd[1]: openqa-continuous-update.service: Main process exited, code=exited, status=4/NOPERMISSION
Nov 11 12:25:35 openqaworker-arm22 systemd[1]: openqa-continuous-update.service: Failed with result 'exit-code'.
Actions
Copy link
 #8

Updated by mkittler 6 months ago

According to https://suse.slack.com/archives/C029APBKLGK/p1699960321972159 there are some infra-related problems due to an ongoing migration. So the problems we saw so are expected to be gone once the migration has settled.

Actions
Copy link
 #9

Updated by okurz 6 months ago

  • Priority changed from Immediate to High

as discussed with mkittler the urgency was resolved so two points to follow up with @mkittler:

  1. Monitor the jobs for the mirroring infrastructure and download repositories and impact on openQA tests
  2. Investigate why packages have been uninstalled in the first place
Actions
Copy link
 #10

Updated by okurz 6 months ago

  • Subject changed from openqaworker-arm22 broken to openqaworker-arm22 broken due to packages automatically removed size:M
  • Description updated (diff)
Actions
Copy link
 #11

Updated by mkittler 6 months ago

The culprit was openqa-auto-update.service:

Nov 10 16:04:51 openqaworker-arm22 systemd[1]: Started Automatically patch system packages. See https://progress.opensuse.org/issues/18006 and https://progress.opensuse.org/issues/70978.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37438]: devel:openQA looks good for Leap 15.5 (aarch64)
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'devel_openQA'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/repositories/devel:openQA/15.5/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'devel_openQA' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'devel_openQA_Leap'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/repositories/devel:openQA:Leap:15.5/15.5/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'devel_openQA_Leap' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'openSUSE-Leap-15.5-1'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/distribution/leap/15.5/repo/oss/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'openSUSE-Leap-15.5-1' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'openSUSE-Leap-15.5-1' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Update repository of openSUSE Backports'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/update/leap/15.5/backports/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Update repository of openSUSE Backports' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Non-OSS Repository'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/distribution/leap/15.5/repo/non-oss/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Non-OSS Repository' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Open H.264 Codec (openSUSE Leap)'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://codecs.opensuse.org/openh264/openSUSE_Leap/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to codecs.opensuse.org port 80 after 4 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Open H.264 Codec (openSUSE Leap)' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Main Repository'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/distribution/leap/15.5/repo/oss/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Main Repository' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Update repository with updates from SUSE Linux Enterprise 15'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/update/leap/15.5/sle/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Update repository with updates from SUSE Linux Enterprise 15' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Main Update Repository'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/update/leap/15.5/oss/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Main Update Repository' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Problem retrieving files from 'Update Repository (Non-Oss)'.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Download (curl) error for 'http://download.opensuse.org/update/leap/15.5/non-oss/repodata/repomd.xml':
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error code: Connection failed
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Error message: Failed to connect to download.opensuse.org port 80 after 5 ms: Couldn't connect to server
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Please see the above error message for a hint.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: Skipping repository 'Update Repository (Non-Oss)' because of the above error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Some of the repositories have not been refreshed because of an error.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Loading repository data...
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Reading installed packages...
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Computing distribution upgrade...
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: The following 382 packages are going to be REMOVED:
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]:   aaa_base-extras acl adjtimex alsa-oss alsa-plugins alsa-plugins-speexrate alsa-plugins-upmix at-spi2-atk-common at-spi2-atk-gtk2 attr audit autofs bind-utils blog breezy busybox-static checkmedia cifs-utils cpio-mt crda cron cronie cups cups-client cu>
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: The following 7 patterns are going to be REMOVED:
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]:   apparmor base documentation enhanced_base minimal_base sw_management yast2_basis
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: 382 packages to remove.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: After the operation, 1.8 GiB will be freed.
Nov 10 16:04:51 openqaworker-arm22 openqa-auto-update[37449]: Continue? [y/n/v/...? shows all options] (y): y
…

So it failed to refresh repos but nevertheless continued.

Actions
Copy link
 #12

Updated by mkittler 6 months ago

That means the problematic command is simply: zypper -n dup --replacefiles --auto-agree-with-licenses --download-in-advance

I also use this myself all the time when updating my local Tumbleweed system - except for the -n flag which is the problematic part. Usually, when packages would be wrongly removed one simply would not proceed.

This is not a new problem. We've observed it before and I have already created a ticket about it (https://github.com/openSUSE/zypper/issues/446).

We have multiple ways to proceed:

  1. Just keep auto-update as-is and live with breakage like this. It doesn't seem to happen very often and in this case the culprit was likely a migration of the repo/mirror infrastructure which doesn't happen every day.
  2. Keep auto-update disabled until the zypper bug has been resolved. This would be very conservative.
    1. Wait until upstream fixes the issue by simply making zypper abort the update if refreshing failed.
    2. Create a PR for zypper ourselves. It shouldn't be too hard.
  3. Call zypper ref manually before (and abort if it fails). Then use --no-refresh on the zypper dup call.

I guess I'll go for option 3. and will check how much effort 2.2 would be (as it would be quite desirable to finally fix this issue at its source).

EDIT: PR for 3: https://github.com/os-autoinst/openQA/pull/5361

Actions
Copy link
 #13

Updated by openqa_review 6 months ago

  • Due date set to 2023-11-29

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #14

Updated by okurz 6 months ago

  • Related to action #150908: o3 "Unable to fetch build results" and "Internal server error" on some pages size:M added
Actions
Copy link
 #15

Updated by mkittler 6 months ago

  • Status changed from In Progress to Resolved

The PR has been merged. As aarch64 jobs look good (and these kinds of incompletes had been restarted automatically) I guess we can consider this ticket resolved. Note that I now actually got some feedback on the upstream issue but we don't need to keep this issue of for tracking this.

Actions
Copy link
 #17

Updated by okurz 6 months ago

  • Related to action #150920: openqaworker-arm22 is unable to join download.opensuse.org in parallel tests = tap mode size:M added
Actions
Copy link
 #18

Updated by okurz 6 months ago

  • Status changed from Resolved to Workable
# zypper dup
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
4 Problems:
Problem: problem with the installed perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64
Problem: problem with the installed perl-Selenium-Remote-Driver-1.49-lp155.2.1.aarch64
Problem: problem with the installed tesseract-ocr-5.3.1-lp155.38.8.aarch64
Problem: problem with the installed libtesseract5-5.3.1-lp155.38.8.aarch64

Problem: problem with the installed perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64
 Solution 1: install perl-Cpanel-JSON-XS-4.36-bp155.2.3.1.aarch64 from vendor openSUSE
  replacing perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64 from vendor obs://build.opensuse.org/devel:openQA
 Solution 2: keep obsolete perl-Cpanel-JSON-XS-4.37-lp155.2.3.aarch64

so not a clean state. Please ensure that the system is up-to-date and that the auto-update and continuous-update service work fine. This might also resolve #150920

Actions
Copy link
 #19

Updated by mkittler 6 months ago

  • Status changed from Workable to Resolved

You just need --allow-vendor-change on your zypper dup line. This is expected and always required in cases when we can remove a package from our devel repo. So not releated to the original issue. I nevertheless just did zypper dup --allow-vendor-change. I guess it makes sense to configure --allow-vendor-change in the zypper config (so I've just did that like we have it in our salt states).

Actions
Copy link

Also available in: Atom PDF