action #104142
osd-deployment pipeline failed: File ... not found on medium
0%
Description
There are several errors in the pipeline:
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749403
Retrieving: monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm [not found] 189 File './noarch/monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/' ... Retrieving: ca-certificates-suse-1.0-lp152.10.1.noarch.rpm [not found] 277 File './noarch/ca-certificates-suse-1.0-lp152.10.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/'
Retry didn't help.
Related issues
History
#3
Updated by okurz 5 months ago
- Blocks action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M added
#4
Updated by okurz 5 months ago
- Status changed from New to In Progress
- Assignee set to okurz
The problem is very likely related to upgrades. For 15.3 the repo is called http://download.suse.de/ibs/NON_Public:/infrastructure/15.3/ , so without the "openSUSE_Leap_" prefix. We don't control NON_Public over salt. Taking a look what I can do
#5
Updated by okurz 5 months ago
salt '*' cmd.run 'zypper --non-interactive se --installed-only --repo NPI || zypper --non-interactive se --installed-only --repo NON_Public_infrastructure'
And I find
openqaworker2.suse.de: Specified repository 'NPI' is disabled. Global option '--plus-content <TAG>' can be used to temporarily enable repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. storage.qa.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker6.suse.de: Specified repository 'NPI' is disabled. Global option '--plus-content <TAG>' can be used to temporarily enable repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker9.suse.de: Specified repository 'NPI' is disabled. Global option '--plus-content <TAG>' can be used to temporarily enable repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker8.suse.de: Specified repository 'NPI' is disabled. Global option '--plus-content <TAG>' can be used to temporarily enable repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker5.suse.de: Specified repository 'NPI' is disabled. Global option '--plus-content <TAG>' can be used to temporarily enable repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker10.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker13.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. malbec.arch.suse.de: Loading repository data... Reading installed packages... No matching items found. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. grenache-1.qa.suse.de: Loading repository data... Reading installed packages... No matching items found. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker-arm-1.suse.de: Loading repository data... Reading installed packages... No matching items found. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker-arm-3.suse.de: Loading repository data... Reading installed packages... S | Name | Summary | Type ---+------------------------------+---------------------------------------------+-------- i+ | ca-certificates-suse | SUSE internal root certificates | package i+ | monitoring-plugins-multipath | Checks multipath connections to SAN storage | package openqaworker-arm-2.suse.de: Loading repository data... Reading installed packages... No matching items found. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqaworker3.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqa-monitor.qa.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. backup.qa.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Repository 'NON_Public_infrastructure' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. openqa.suse.de: Repository 'NPI' not found by its alias, number, or URI. Use 'zypper repos' to get the list of defined repositories. Loading repository data... Reading installed packages... S | Name | Summary | Type ---+----------------------+---------------------------------+-------- i+ | ca-certificates-suse | SUSE internal root certificates | package i+ | check_mk-agent | Agent for check_mk | package
# salt '*' cmd.run 'sed -i "s/openSUSE_Leap_//" /etc/zypp/repos.d/{NPI,NON_Public_infrastructure}.repo' storage.qa.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker3.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker2.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker8.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker6.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker5.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker9.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory malbec.arch.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory grenache-1.qa.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory backup.qa.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker10.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqa-monitor.qa.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker13.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqa.suse.de: sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory openqaworker-arm-1.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker-arm-3.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory openqaworker-arm-2.suse.de: sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory ERROR: Minions returned with non-zero exit code
Now
salt '*' cmd.run 'zypper ref'
#6
Updated by okurz 5 months ago
- Status changed from In Progress to Feedback
Retriggered OSD deployment job https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749445
Deployment continued https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/277889
So we have the package "ca-certificates-suse" installed on all openQA server+worker but on some machines that comes from NPI, instead of SUSE_CA which is a repo we install on all machines over salt.
To make sure this is consistent I did
salt '*' cmd.run 'zypper in --allow-vendor-change -f -r SUSE_CA ca-certificates-suse' salt '*' cmd.run 'zypper --non-interactive mr -d NPI || zypper --non-interactive mr -d NON_Public_infrastructure'
so now we have ca-certificates-suse installed consistently from SUSE_CA and non-salted repos disabled. Now this leaves at least "check_mk-agent" as orphan on OSD.
We should consider using https://github.com/okurz/scripts/blob/master/zypper-rm-orphaned everywhere, or at least zypper packages --orphaned| awk '/^i/{ print $5 }' ORS=" "
in a first check to check for orphans:
# salt '*' cmd.run 'zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" "' openqaworker3.suse.de: System management is locked by the application with pid 10835 (zypper). Close this application before trying again. backup.qa.suse.de: System management is locked by the application with pid 29380 (zypper). Close this application before trying again. openqa-monitor.qa.suse.de: System management is locked by the application with pid 25012 (zypper). Close this application before trying again. storage.qa.suse.de: libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses14 libyui14 lifecycle-data-sle-module-basesystem lifecycle-data-sle-module-desktop-applications lifecycle-data-sle-module-desktop-productivity lifecycle-data-sle-module-hpc lifecycle-data-sle-module-server-applications zypper-search-packages-plugin openqaworker2.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin openqaworker9.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin openqaworker8.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin openqaworker6.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin openqaworker5.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin grenache-1.qa.suse.de: PackageHub-release PackageHub-release-POOL bolt check_mk-agent facter libcdio14 libcdio_cdda1 libcdio_paranoia1 libffi4 libruby2_1-2_1 lifecycle-data-sle-module-toolchain monitoring-plugins-multipath puppet ruby2.1 ruby2.1-rubygem-hiera-1 ruby2.1-rubygem-json_pure ruby2.1-rubygem-ruby-shadow ruby2.1-stdlib rubygem-hiera-1 sle-module-adv-systems-management-release-POOL sle-module-toolchain-release-POOL zypper-search-packages-plugin openqaworker10.suse.de: libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 zypper-search-packages-plugin openqaworker13.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin openqa.suse.de: check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses11 libyui11 malbec.arch.suse.de: check_mk-agent monitoring-plugins-multipath openqaworker-arm-3.suse.de: monitoring-plugins-multipath openqaworker-arm-2.suse.de: Minion did not return. [No response] The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command: salt-run jobs.lookup_jid 20211217095514906830 openqaworker-arm-1.suse.de: Minion did not return. [No response] The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command: salt-run jobs.lookup_jid 20211217095514906830 ERROR: Minions returned with non-zero exit code
looks ok to me (except for the non-responsive or busy machines). So I asked in https://suse.slack.com/archives/C029APBKLGK/p1639735037085000 if we still need this package multipath monitoring, e.g. for icinga monitoring, but did go ahead with
salt '*' cmd.run 'zypper -n rm $(zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" ")'