Project

General

Profile

action #104142

osd-deployment pipeline failed: File ... not found on medium

Added by tinita 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
okurz
Target version:
openQA Project - Ready
Start date:
2021-12-17
Due date:
% Done:

0%

Estimated time:

Description

There are several errors in the pipeline:

https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749403

Retrieving: monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm [not found]
189    File './noarch/monitoring-plugins-multipath-1.12-lp152.106.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/'
...
Retrieving: ca-certificates-suse-1.0-lp152.10.1.noarch.rpm [not found]
277    File './noarch/ca-certificates-suse-1.0-lp152.10.1.noarch.rpm' not found on medium 'http://download.suse.de/ibs/NON_Public:/infrastructure/openSUSE_Leap_15.3/'

Retry didn't help.

Related issues

Blocks openQA Infrastructure - action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:MResolved

History

#1 Updated by tinita 5 months ago

  • Description updated (diff)

#2 Updated by okurz 5 months ago

  • Priority changed from High to Urgent

As it's blocking deployment I consider this urgent then

#3 Updated by okurz 5 months ago

  • Blocks action #99192: Upgrade osd workers and openqa-monitor to openSUSE Leap 15.3 size:M added

#4 Updated by okurz 5 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz

The problem is very likely related to upgrades. For 15.3 the repo is called http://download.suse.de/ibs/NON_Public:/infrastructure/15.3/ , so without the "openSUSE_Leap_" prefix. We don't control NON_Public over salt. Taking a look what I can do

#5 Updated by okurz 5 months ago 

salt '*' cmd.run 'zypper --non-interactive se --installed-only --repo NPI ||  zypper --non-interactive se --installed-only --repo NON_Public_infrastructure'

And I find

openqaworker2.suse.de:
    Specified repository 'NPI' is disabled.
    Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
storage.qa.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker6.suse.de:
    Specified repository 'NPI' is disabled.
    Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker9.suse.de:
    Specified repository 'NPI' is disabled.
    Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker8.suse.de:
    Specified repository 'NPI' is disabled.
    Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker5.suse.de:
    Specified repository 'NPI' is disabled.
    Global option '--plus-content <TAG>' can be used to temporarily enable repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker10.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker13.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
malbec.arch.suse.de:
    Loading repository data...
    Reading installed packages...
    No matching items found.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
grenache-1.qa.suse.de:
    Loading repository data...
    Reading installed packages...
    No matching items found.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker-arm-1.suse.de:
    Loading repository data...
    Reading installed packages...
    No matching items found.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker-arm-3.suse.de:
    Loading repository data...
    Reading installed packages...

    S  | Name                         | Summary                                     | Type
    ---+------------------------------+---------------------------------------------+--------
    i+ | ca-certificates-suse         | SUSE internal root certificates             | package
    i+ | monitoring-plugins-multipath | Checks multipath connections to SAN storage | package
openqaworker-arm-2.suse.de:
    Loading repository data...
    Reading installed packages...
    No matching items found.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqaworker3.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqa-monitor.qa.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
backup.qa.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Repository 'NON_Public_infrastructure' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
openqa.suse.de:
    Repository 'NPI' not found by its alias, number, or URI.
    Use 'zypper repos' to get the list of defined repositories.
    Loading repository data...
    Reading installed packages...

    S  | Name                 | Summary                         | Type
    ---+----------------------+---------------------------------+--------
    i+ | ca-certificates-suse | SUSE internal root certificates | package
    i+ | check_mk-agent       | Agent for check_mk              | package

# salt '*' cmd.run 'sed -i "s/openSUSE_Leap_//" /etc/zypp/repos.d/{NPI,NON_Public_infrastructure}.repo'
storage.qa.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker3.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker2.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker8.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker6.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker5.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker9.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
malbec.arch.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
grenache-1.qa.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
backup.qa.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker10.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqa-monitor.qa.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker13.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqa.suse.de:
    sed: can't read /etc/zypp/repos.d/NPI.repo: No such file or directory
openqaworker-arm-1.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker-arm-3.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
openqaworker-arm-2.suse.de:
    sed: can't read /etc/zypp/repos.d/NON_Public_infrastructure.repo: No such file or directory
ERROR: Minions returned with non-zero exit code

Now

salt '*' cmd.run 'zypper ref'

#6 Updated by okurz 5 months ago

  • Status changed from In Progress to Feedback

Retriggered OSD deployment job https://gitlab.suse.de/openqa/osd-deployment/-/jobs/749445

Deployment continued https://gitlab.suse.de/openqa/osd-deployment/-/pipelines/277889

So we have the package "ca-certificates-suse" installed on all openQA server+worker but on some machines that comes from NPI, instead of SUSE_CA which is a repo we install on all machines over salt.

To make sure this is consistent I did

salt '*' cmd.run 'zypper in --allow-vendor-change -f -r SUSE_CA ca-certificates-suse'
salt '*' cmd.run 'zypper --non-interactive mr -d NPI ||  zypper --non-interactive mr -d NON_Public_infrastructure'

so now we have ca-certificates-suse installed consistently from SUSE_CA and non-salted repos disabled. Now this leaves at least "check_mk-agent" as orphan on OSD.

We should consider using https://github.com/okurz/scripts/blob/master/zypper-rm-orphaned everywhere, or at least zypper packages --orphaned| awk '/^i/{ print $5 }' ORS=" " in a first check to check for orphans:

# salt '*' cmd.run 'zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" "'
openqaworker3.suse.de:
    System management is locked by the application with pid 10835 (zypper).
    Close this application before trying again.
backup.qa.suse.de:
    System management is locked by the application with pid 29380 (zypper).
    Close this application before trying again.
openqa-monitor.qa.suse.de:
    System management is locked by the application with pid 25012 (zypper).
    Close this application before trying again.
storage.qa.suse.de:
    libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses14 libyui14 lifecycle-data-sle-module-basesystem lifecycle-data-sle-module-desktop-applications lifecycle-data-sle-module-desktop-productivity lifecycle-data-sle-module-hpc lifecycle-data-sle-module-server-applications zypper-search-packages-plugin
openqaworker2.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker9.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker8.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker6.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqaworker5.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
grenache-1.qa.suse.de:
    PackageHub-release PackageHub-release-POOL bolt check_mk-agent facter libcdio14 libcdio_cdda1 libcdio_paranoia1 libffi4 libruby2_1-2_1 lifecycle-data-sle-module-toolchain monitoring-plugins-multipath puppet ruby2.1 ruby2.1-rubygem-hiera-1 ruby2.1-rubygem-json_pure ruby2.1-rubygem-ruby-shadow ruby2.1-stdlib rubygem-hiera-1 sle-module-adv-systems-management-release-POOL sle-module-toolchain-release-POOL zypper-search-packages-plugin
openqaworker10.suse.de:
    libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 zypper-search-packages-plugin
openqaworker13.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libyui-ncurses11 libyui11 monitoring-plugins-multipath zypper-search-packages-plugin
openqa.suse.de:
    check_mk-agent libbind9-160 libdns169 libirs160 libisc166 libisccc160 libisccfg160 liblwres160 libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 libyui-ncurses11 libyui11
malbec.arch.suse.de:
    check_mk-agent monitoring-plugins-multipath
openqaworker-arm-3.suse.de:
    monitoring-plugins-multipath
openqaworker-arm-2.suse.de:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20211217095514906830
openqaworker-arm-1.suse.de:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20211217095514906830
ERROR: Minions returned with non-zero exit code

looks ok to me (except for the non-responsive or busy machines). So I asked in https://suse.slack.com/archives/C029APBKLGK/p1639735037085000 if we still need this package multipath monitoring, e.g. for icinga monitoring, but did go ahead with

salt '*' cmd.run 'zypper -n rm $(zypper packages --orphaned| awk "/^i/{ print \$5 }" ORS=" ")'

#7 Updated by okurz 5 months ago

  • Status changed from Feedback to Resolved

icinga/thruk monitoring might be impacted by the removed stuff but rest looks fine so I consider this done

Also available in: Atom PDF