Project

General

Profile

Actions

action #127907

closed

openQA Project (public) - coordination #127031: [saga][epic] openQA for SUSE customers

openQA Project (public) - coordination #127910: [epic] openQA in SLE modules

jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time size:M

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Start date:
2023-04-19
Due date:
% Done:

0%

Estimated time:

Description

Observation

http://jenkins.qa.suse.de/manage/ mentions that there is a new version of jenkins. In the past the jenkins package was automatically updated. zypper dup shows some pending changes on the host even though the "auto-update.service" is enabled. The situation should be cross-checked and fixed.

Acceptance criteria

  • AC1: packages on jenkins.qa.suse.de including "jenkins" are automatically upgraded
  • AC2: The problem source is understood and prevented for the future and other hosts

Suggestions

  • jenkins.qa.suse.de should be salt-controlled from https://gitlab.suse.de/openqa/salt-states-openqa and hence should have automatic upgrades and monitoring. It should be cross-checked why this did not have the expected effect
  • Ensure that all generic workers are properly upgraded
  • Add automatic upgrade of all machines from all repos in salt

Further details

I came across the current status on jenkins.qa.suse.de as I wanted to review the current state of the automatic submissions of openQA related packages from devel:openQA to openSUSE:Factory as we are planning to apply an equivalent approach for automatic submission to SLE for QAaaS

Actions #1

Updated by okurz over 1 year ago

  • Priority changed from Normal to High
Actions #2

Updated by okurz over 1 year ago

  • Tags changed from infra, jenkins, update, upgrade, salt to infra, jenkins, update, upgrade, salt, qaaas
  • Description updated (diff)
  • Parent task set to #127910
Actions #3

Updated by okurz over 1 year ago

mkittler and me looked into the problem and realized that we have a bigger conceptual problem. jenkins.qa.suse.de is a "generic" host managed by https://gitlab.suse.de/openqa/salt-states-openqa among others. We assumed that package upgrades would be handled by https://gitlab.suse.de/openqa/salt-states-openqa/-/tree/master/auto-update however that auto-update only applies patches so a new package in the repository https://pkg.origin.jenkins.io/opensuse-stable/ is never picked up. We don't use zypper dup in auto-update as we want to explicitly deploy openQA related packages in https://gitlab.suse.de/openqa/osd-deployment/ so we need to find a solution for all non-openQA-generic hosts to call zypper dup there.

With

sudo salt --no-color -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n dup --dry-run --details'

we manually checked how package upgrades would look like and it turned out to be a bigger list of pending changes and maybe also wrong repository priorities and such.

  • TODO go over list of pending changes on all generic machines and ensure an updated state
  • TODO in salt recipes add auto-upgrade service for all non-openQA machines
Actions #4

Updated by okurz over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to okurz
Actions #5

Updated by openqa_review over 1 year ago

  • Due date set to 2023-05-06

Setting due date based on mean cycle time of SUSE QE Tools

Actions #6

Updated by okurz over 1 year ago

From OSD I did

sudo salt --no-color --state-output=changes -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n patch'

and then

sudo salt --no-color --state-output=changes -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n patch --with-optional'

as some patches were listed as not being installed. Only machine with a bigger problem was qamaster.qa.suse.de:

    Warning: Patch 'openSUSE-SLE-15.4-2023-433-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-3999-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-848-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-464-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-201-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-1897-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-1710-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-149-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-4585-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-4072-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-4007-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-4006-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-3844-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2022-3806-1' is interactive, skipping.
    Warning: Patch 'openSUSE-SLE-15.4-2023-169-1' is interactive, skipping.
    Resolving package dependencies...
    6 Problems:
    Problem: the to be installed patch:openSUSE-SLE-15.4-2023-161-1.noarch conflicts with 'python-py.noarch < 1.10.0-150100.5.12.1' provided by the installed python-py-1.8.1-11.12.4.noarch
    Problem: the to be installed patch:openSUSE-SLE-15.4-2022-3022-1.noarch conflicts with 'python-pyOpenSSL.noarch < 21.0.0-150400.3.3.1' provided by the installed python-pyOpenSSL-17.1.0-4.23.1.noarch
    Problem: the to be installed patch:openSUSE-SLE-15.4-2023-159-1.noarch conflicts with 'python-setuptools.noarch < 44.1.1-150400.3.3.1' provided by the installed python-setuptools-40.6.2-4.18.1.noarch
    Problem: the to be installed patch:openSUSE-SLE-15.4-2022-3600-1.noarch conflicts with 'python-urlgrabber.noarch < 4.1.0-150400.4.6.1' provided by the installed python-urlgrabber-3.9.1-15.2.noarch
    Problem: the installed python-requests-2.24.0-8.11.4.noarch requires 'python-py', but this requirement cannot be provided
    Problem: the to be installed patch:openSUSE-SLE-15.4-2022-4262-1.noarch conflicts with 'libdevmapper1_03-32bit.x86_64 < 2.03.05_1.02.163-150400.185.1' provided by the installed libdevmapper1_03-32bit-1.02.163-150400.178.1.x86_64

    Problem: the to be installed patch:openSUSE-SLE-15.4-2023-161-1.noarch conflicts with 'python-py.noarch < 1.10.0-150100.5.12.1' provided by the installed python-py-1.8.1-11.12.4.noarch
     Solution 1: deinstallation of python-py-1.8.1-11.12.4.noarch
     Solution 2: do not install patch:openSUSE-SLE-15.4-2023-161-1.noarch

    Choose from above solutions by number or skip, retry or cancel [1/2/s/r/c/d/?] (c): c

so I manually logged in and selected solutions for the problem by deinstalling some python packages and concluded the patching.

On baremetal-support the openQA package wouldn't upgrade due to conflicting user IDs regarding geekotest as there is an NFS mount used.

So I changed user ids and updated ownership:

systemctl stop postgresql openqa-webui openqa-scheduler
usermod -u 17256 rbrown
usermod -u 1001 geekotest
umount /var/lib/openqa/share/factory/iso
chown -R geekotest /var/lib/openqa/
find / -uid 483 -exec chown -v -h 1001 '{}' \;
zypper dup
systemctl start default
mount -a
Actions #7

Updated by okurz over 1 year ago

  • Subject changed from jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time to jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time size:M
  • Description updated (diff)
Actions #8

Updated by okurz over 1 year ago

  • Due date deleted (2023-05-06)
  • Status changed from In Progress to Resolved

On openqa-piworker I changed priorities for repositories and upgraded, some more packages from devel:python:backports and now it looks good.
I merged https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/846 . It was deployed cleanly and salt state looks good as well. All generic hosts including jenkins.qa.suse.de are now up-to-date and should be kept as such.

Actions

Also available in: Atom PDF