Project

General

Profile

Actions

action #131540

closed

openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M

Added by okurz 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/repos.sls#L62 does

/etc/zypp/zypp.conf:
  ini.options_present:
    - sections:
        main:
          solver.dupAllowVendorChange: true

but openqa-piworker:/home/okurz # grep -i vendor /etc/zypp/zypp* says

…
/etc/zypp/zypp.conf:solver.dupAllowVendorChange = false

Acceptance criteria

  • AC1: all salt controlled generic machines, e.g. openqa-piworker, allow vendor change same as openQA workers
  • AC2: the change is supplied by salt
  • AC3: All OSD salt controlled machines are up-to-date

Suggestions

  • Convince yourself that "AllowVendorChange=true" is what we want for all machines. If not convinced yet then watch https://www.youtube.com/watch?v=x8kEaJU6hlw . If still not convinced then talk to okurz.
  • Carefully manually check every generic machine how the effect would be, e.g. with salt -C 'not G@roles:webui and not G@roles:worker' cmd.run 'zypper -n dup --dry-run --allow-vendor-change'
  • Move the change from openqa/repos.sls to a file that is applied on all machines
  • Ensure that all our salt controlled machines are up-to-date after the above change (if any), e.g. salt \* cmd.run 'zypper -n dup'

Related issues 1 (0 open1 closed)

Copied from openQA Infrastructure - action #131249: [alert][ci][deployment] OSD deployment failed, grenache-1, worker5, worker2 salt-minion does not return, error message "No response" size:MResolvedokurz2023-06-22

Actions
Actions #1

Updated by okurz 10 months ago

  • Copied from action #131249: [alert][ci][deployment] OSD deployment failed, grenache-1, worker5, worker2 salt-minion does not return, error message "No response" size:M added
Actions #2

Updated by tinita 10 months ago

Is it related to the high disk usage on that machine?

Actions #3

Updated by okurz 10 months ago

Yes, likely. dheidler and me conducted a zypper dup which went fine but apparently at least during the upgrade process that took some more space. The upgrade concluded but the available+used space should be revisited.

Actions #4

Updated by nicksinger 10 months ago

we had data below the nfs-mount "/var/lib/openqa/share". I found them by bind-mounting / into a different directory like so: mount -o bind / /tmp/mnt/
In that directory I could free up ~6G and the alert is "Resolved" again.

Actions #5

Updated by okurz 10 months ago

  • Subject changed from openqa-piworker fails to upgrade many packages. Seems like vendor change is not enabled though our salt states should do that to openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines
  • Description updated (diff)
Actions #6

Updated by okurz 10 months ago

  • Subject changed from openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines to openqa-piworker fails to upgrade many packages. vendor change is not enabled as our salt states so far only do that for openQA machines, not generic machines size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #7

Updated by mkittler 10 months ago

  • Assignee set to mkittler
Actions #8

Updated by mkittler 10 months ago

  • Status changed from Workable to In Progress

salt -C 'not G@roles:webui and not G@roles:worker' cmd.run 'zypper -n dup --dry-run --allow-vendor-change' looks good; no package would actually change its vendor right now.

Actions #9

Updated by mkittler 10 months ago

Actions #10

Updated by mkittler 10 months ago

  • Status changed from In Progress to Feedback

It seems to work:

martchus@openqa-monitor:~> grep -i vendorchange /etc/zypp/zypp.conf
# solver.allowVendorChange = false
## you may indeed benefit from disabling the VendorChange. Packages from
solver.dupAllowVendorChange = false
martchus@openqa:/srv/salt> sudo salt -C 'G@nodename:openqa-monitor' -l error --state-output=changes state.apply
…
Summary for openqa-monitor.qa.suse.de
--------------
Succeeded: 358 (changed=4)
Failed:      0
--------------
Total states run:     358
Total run time:    76.573 s
martchus@openqa-monitor:~> grep -i vendorchange /etc/zypp/zypp.conf
# solver.allowVendorChange = false
## you may indeed benefit from disabling the VendorChange. Packages from
solver.dupAllowVendorChange = True
martchus@openqa-monitor:~> sudo zypper dup
Retrieving repository 'openSUSE-Leap-15.4-Update' metadata ...........................................................................................................................................................................................................................................................[done]
Building repository 'openSUSE-Leap-15.4-Update' cache ................................................................................................................................................................................................................................................................[done]
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...

The following 6 packages are going to be upgraded:
  libzypp yast2-storage-ng zypper zypper-aptitude zypper-log zypper-needs-restarting

6 packages to upgrade.
Overall download size: 5.1 MiB. Already cached: 0 B. After the operation, additional 4.0 KiB will be used.
Continue? [y/n/v/...? shows all options] (y):
…
Actions #11

Updated by okurz 10 months ago

  • Status changed from Feedback to Resolved

I crosschecked once more with

salt --no-color \* cmd.run 'grep -i vendorchange /etc/zypp/zypp.conf'
salt --no-color \* cmd.run 'zypper -n dup --details --dry-run'

and the situation looks good now.

Actions

Also available in: Atom PDF