Actions
action #104142
closedosd-deployment pipeline failed: File ... not found on medium
Added by tinita over 2 years ago. Updated over 2 years ago.
Start date:
2021-12-17
Due date:
% Done:
0%
Estimated time:
Description
There are several errors in the pipeline:
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749403
Retrieving: monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm [not found]
189 File './noarch/monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/'
...
Retrieving: ca-certificates-suse-1.0-lp152.10.1.noarch.rpm [not found]
277 File './noarch/ca-certificates-suse-1.0-lp152.10.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/'
Retry didn't help.
Updated by okurz over 2 years ago
- Priority changed from High to Urgent
As it's blocking deployment I consider this urgent then
Updated by okurz over 2 years ago
- Blocks action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M added
Updated by okurz over 2 years ago
- Status changed from New to In Progress
- Assignee set to okurz
The problem is very likely related to upgrades. For 15.3 the repo is called http://download.suse.de/ibs/NON_Public:/infrastructure/15.3/ , so without the "openSUSE_Leap_" prefix. We don't control NON_Public over salt. Taking a look what I can do
Updated by okurz over 2 years ago
salt '*' cmd.run 'zypper --non-interactive se --installed-only --repo NPI || zypper --non-interactive se --installed-only --repo NON_Public_infrastructure'
And I find
openqaworker2.suse.de:
Specified repository 'NPI' is disabled.
Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
storage.qa.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker6.suse.de:
Specified repository 'NPI' is disabled.
Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker9.suse.de:
Specified repository 'NPI' is disabled.
Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker8.suse.de:
Specified repository 'NPI' is disabled.
Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker5.suse.de:
Specified repository 'NPI' is disabled.
Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker10.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker13.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
malbec.arch.suse.de:
Loading repository data...
Reading installed packages...
No matching items found.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
grenache-1.qa.suse.de:
Loading repository data...
Reading installed packages...
No matching items found.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker-arm-1.suse.de:
Loading repository data...
Reading installed packages...
No matching items found.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker-arm-3.suse.de:
Loading repository data...
Reading installed packages...
S | Name | Summary | Type
---+------------------------------+---------------------------------------------+--------
i+ | ca-certificates-suse | SUSE internal root certificates | package
i+ | monitoring-plugins-multipath | Checks multipath connections to SAN storage | package
openqaworker-arm-2.suse.de:
Loading repository data...
Reading installed packages...
No matching items found.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqaworker3.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqa-monitor.qa.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
backup.qa.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
openqa.suse.de:
Repository 'NPI' not found by its alias, number, or URI.
Use 'zypper repos' to get the list of defined repositories.
Loading repository data...
Reading installed packages...
S | Name | Summary | Type
---+----------------------+---------------------------------+--------
i+ | ca-certificates-suse | SUSE internal root certificates | package
i+ | check_mk-agent | Agent for check_mk | package
# salt '*' cmd.run 'sed -i "s/openSUSE_Leap_//" /etc/zypp/repos.d/{NPI,NON_Public_infrastructure}.repo'
storage.qa.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker3.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker2.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker8.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker6.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker5.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker9.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
malbec.arch.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
grenache-1.qa.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
backup.qa.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker10.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqa-monitor.qa.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker13.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqa.suse.de:
sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
openqaworker-arm-1.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker-arm-3.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker-arm-2.suse.de:
sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
ERROR: Minions returned with non-zero exit code
Now
salt '*' cmd.run 'zypper ref'
Updated by okurz over 2 years ago
- Status changed from In Progress to Feedback
Retriggered OSD deployment job https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749445
Deployment continued https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/277889
So we have the package "ca-certificates-suse" installed on all openQA server+worker but on some machines that comes from NPI, instead of SUSE_CA which is a repo we install on all machines over salt.
To make sure this is consistent I did
salt '*' cmd.run 'zypper in --allow-vendor-change -f -r SUSE_CA ca-certificates-suse'
salt '*' cmd.run 'zypper --non-interactive mr -d NPI || zypper --non-interactive mr -d NON_Public_infrastructure'
so now we have ca-certificates-suse installed consistently from SUSE_CA and non-salted repos disabled. Now this leaves at least "check_mk-agent" as orphan on OSD.
We should consider using https://github.com/okurz/scripts/blob/master/zypper-rm-orphaned everywhere, or at least zypper packages --orphaned| awk '/^i/{ print $5 }' ORS=" " in a first check to check for orphans:
# salt '*' cmd.run 'zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" "'
openqaworker3.suse.de:
System management is locked by the application with pid 10835 (zypper).
Close this application before trying again.
backup.qa.suse.de:
System management is locked by the application with pid 29380 (zypper).
Close this application before trying again.
openqa-monitor.qa.suse.de:
System management is locked by the application with pid 25012 (zypper).
Close this application before trying again.
storage.qa.suse.de:
libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses14 libyui14 lifecycle-data-sle-module-basesystem lifecycle-data-sle-module-desktop-applications lifecycle-data-sle-module-desktop-productivity lifecycle-data-sle-module-hpc lifecycle-data-sle-module-server-applications zypper-search-packages-plugin
openqaworker2.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker9.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker8.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker6.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker5.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
grenache-1.qa.suse.de:
PackageHub-release PackageHub-release-POOL bolt check_mk-agent facter libcdio14 libcdio_cdda1 libcdio_paranoia1 libffi4 libruby2_1-2_1 lifecycle-data-sle-module-toolchain monitoring-plugins-multipath puppet ruby2.1 ruby2.1-rubygem-hiera-1 ruby2.1-rubygem-json_pure ruby2.1-rubygem-ruby-shadow ruby2.1-stdlib rubygem-hiera-1 sle-module-adv-systems-management-release-POOL sle-module-toolchain-release-POOL zypper-search-packages-plugin
openqaworker10.suse.de:
libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 zypper-search-packages-plugin
openqaworker13.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqa.suse.de:
check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses11 libyui11
malbec.arch.suse.de:
check_mk-agent monitoring-plugins-multipath
openqaworker-arm-3.suse.de:
monitoring-plugins-multipath
openqaworker-arm-2.suse.de:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20211217095514906830
openqaworker-arm-1.suse.de:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20211217095514906830
ERROR: Minions returned with non-zero exit code
looks ok to me (except for the non-responsive or busy machines). So I asked in https://suse.slack.com/archives/C029APBKLGK/p1639735037085000 if we still need this package multipath monitoring, e.g. for icinga monitoring, but did go ahead with
salt '*' cmd.run 'zypper -n rm $(zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" ")'
Updated by okurz over 2 years ago
- Status changed from Feedback to Resolved
icinga/thruk monitoring might be impacted by the removed stuff but rest looks fine so I consider this done
Actions