action #131540
closed
openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M
Added by okurz 11 months ago.
Updated 11 months ago.
Description
Observation¶
https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/repos.sls#L62 does
/etc/zypp/zypp.conf:
ini.options_present:
- sections:
main:
solver.dupAllowVendorChange: true
but openqa-piworker:/home/okurz # grep -i vendor /etc/zypp/zypp* says
…
/etc/zypp/zypp.conf:solver.dupAllowVendorChange = false
Acceptance criteria¶
- AC1: all salt controlled generic machines, e.g. openqa-piworker, allow vendor change same as openQA workers
- AC2: the change is supplied by salt
- AC3: All OSD salt controlled machines are up-to-date
Suggestions¶
- Convince yourself that "AllowVendorChange=true" is what we want for all machines. If not convinced yet then watch https://www.youtube.com/watch?v=x8kEaJU6hlw . If still not convinced then talk to okurz.
- Carefully manually check every generic machine how the effect would be, e.g. with
salt -C 'not G@roles:webui and not G@roles:worker' cmd.run 'zypper -n dup --dry-run --allow-vendor-change'
- Move the change from openqa/repos.sls to a file that is applied on all machines
- Ensure that all our salt controlled machines are up-to-date after the above change (if any), e.g.
salt \* cmd.run 'zypper -n dup'
- Copied from action #131249: [alert][ci][deployment] OSD deployment failed, grenache-1, worker5, worker2 salt-minion does not return, error message "No response" size:M added
Is it related to the high disk usage on that machine?
Yes, likely. dheidler and me conducted a zypper dup which went fine but apparently at least during the upgrade process that took some more space. The upgrade concluded but the available+used space should be revisited.
we had data below the nfs-mount "/var/lib/openqa/share". I found them by bind-mounting / into a different directory like so: mount -o bind / /tmp/mnt/
In that directory I could free up ~6G and the alert is "Resolved" again.
- Subject changed from openqa-piworker fails to upgrade many packages. Seems like vendor change is not enabled though our salt states should do that to openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines
- Description updated (diff)
- Subject changed from openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines to openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
salt -C 'not G@roles:webui and not G@roles:worker' cmd.run 'zypper -n dup --dry-run --allow-vendor-change' looks good; no package would actually change its vendor right now.
- Status changed from In Progress to Feedback
It seems to work:
martchus@openqa-monitor:~> grep -i vendorchange /etc/zypp/zypp.conf
# solver.allowVendorChange = false
## you may indeed benefit from disabling the VendorChange. Packages from
solver.dupAllowVendorChange = false
martchus@openqa:/srv/salt> sudo salt -C 'G@nodename:openqa-monitor' -l error --state-output=changes state.apply
…
Summary for openqa-monitor.qa.suse.de
--------------
Succeeded: 358 (changed=4)
Failed: 0
--------------
Total states run: 358
Total run time: 76.573 s
martchus@openqa-monitor:~> grep -i vendorchange /etc/zypp/zypp.conf
# solver.allowVendorChange = false
## you may indeed benefit from disabling the VendorChange. Packages from
solver.dupAllowVendorChange = True
martchus@openqa-monitor:~> sudo zypper dup
Retrieving repository 'openSUSE-Leap-15.4-Update' metadata ...........................................................................................................................................................................................................................................................[done]
Building repository 'openSUSE-Leap-15.4-Update' cache ................................................................................................................................................................................................................................................................[done]
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
The following 6 packages are going to be upgraded:
libzypp yast2-storage-ng zypper zypper-aptitude zypper-log zypper-needs-restarting
6 packages to upgrade.
Overall download size: 5.1 MiB. Already cached: 0 B. After the operation, additional 4.0 KiB will be used.
Continue? [y/n/v/...? shows all options] (y):
…
- Status changed from Feedback to Resolved
I crosschecked once more with
salt --no-color \* cmd.run 'grep -i vendorchange /etc/zypp/zypp.conf'
salt --no-color \* cmd.run 'zypper -n dup --details --dry-run'
and the situation looks good now.
Also available in: Atom
PDF