action #127907
closedopenQA Project - coordination #127031: [saga][epic] openQA for SUSE customers
openQA Project - coordination #127910: [epic] openQA in SLE modules
jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time size:M
0%
Description
Observation¶
http://jenkins.qa.suse.de/manage/ mentions that there is a new version of jenkins. In the past the jenkins package was automatically updated. zypper dup
shows some pending changes on the host even though the "auto-update.service" is enabled. The situation should be cross-checked and fixed.
Acceptance criteria¶
- AC1: packages on jenkins.qa.suse.de including "jenkins" are automatically upgraded
- AC2: The problem source is understood and prevented for the future and other hosts
Suggestions¶
- jenkins.qa.suse.de should be salt-controlled from https://gitlab.suse.de/openqa/salt-states-openqa and hence should have automatic upgrades and monitoring. It should be cross-checked why this did not have the expected effect
- Ensure that all generic workers are properly upgraded
- Add automatic upgrade of all machines from all repos in salt
Further details¶
I came across the current status on jenkins.qa.suse.de as I wanted to review the current state of the automatic submissions of openQA related packages from devel:openQA to openSUSE:Factory as we are planning to apply an equivalent approach for automatic submission to SLE for QAaaS
Updated by okurz over 1 year ago
- Tags changed from infra, jenkins, update, upgrade, salt to infra, jenkins, update, upgrade, salt, qaaas
- Description updated (diff)
- Parent task set to #127910
Updated by okurz over 1 year ago
mkittler and me looked into the problem and realized that we have a bigger conceptual problem. jenkins.qa.suse.de is a "generic" host managed by https://gitlab.suse.de/openqa/salt-states-openqa among others. We assumed that package upgrades would be handled by https://gitlab.suse.de/openqa/salt-states-openqa/-/tree/master/auto-update however that auto-update only applies patches so a new package in the repository https://pkg.origin.jenkins.io/opensuse-stable/ is never picked up. We don't use zypper dup
in auto-update as we want to explicitly deploy openQA related packages in https://gitlab.suse.de/openqa/osd-deployment/ so we need to find a solution for all non-openQA-generic hosts to call zypper dup there.
With
sudo salt --no-color -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n dup --dry-run --details'
we manually checked how package upgrades would look like and it turned out to be a bigger list of pending changes and maybe also wrong repository priorities and such.
- TODO go over list of pending changes on all generic machines and ensure an updated state
- TODO in salt recipes add auto-upgrade service for all non-openQA machines
Updated by okurz over 1 year ago
- Status changed from New to In Progress
- Assignee set to okurz
Updated by openqa_review over 1 year ago
- Due date set to 2023-05-06
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz over 1 year ago
From OSD I did
sudo salt --no-color --state-output=changes -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n patch'
and then
sudo salt --no-color --state-output=changes -C 'not G@roles:worker and not G@roles:webui' cmd.run 'zypper -n patch --with-optional'
as some patches were listed as not being installed. Only machine with a bigger problem was qamaster.qa.suse.de:
Warning: Patch 'openSUSE-SLE-15.4-2023-433-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-3999-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-848-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-464-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-201-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-1897-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-1710-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-149-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-4585-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-4072-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-4007-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-4006-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-3844-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2022-3806-1' is interactive, skipping.
Warning: Patch 'openSUSE-SLE-15.4-2023-169-1' is interactive, skipping.
Resolving package dependencies...
6 Problems:
Problem: the to be installed patch:openSUSE-SLE-15.4-2023-161-1.noarch conflicts with 'python-py.noarch < 1.10.0-150100.5.12.1' provided by the installed python-py-1.8.1-11.12.4.noarch
Problem: the to be installed patch:openSUSE-SLE-15.4-2022-3022-1.noarch conflicts with 'python-pyOpenSSL.noarch < 21.0.0-150400.3.3.1' provided by the installed python-pyOpenSSL-17.1.0-4.23.1.noarch
Problem: the to be installed patch:openSUSE-SLE-15.4-2023-159-1.noarch conflicts with 'python-setuptools.noarch < 44.1.1-150400.3.3.1' provided by the installed python-setuptools-40.6.2-4.18.1.noarch
Problem: the to be installed patch:openSUSE-SLE-15.4-2022-3600-1.noarch conflicts with 'python-urlgrabber.noarch < 4.1.0-150400.4.6.1' provided by the installed python-urlgrabber-3.9.1-15.2.noarch
Problem: the installed python-requests-2.24.0-8.11.4.noarch requires 'python-py', but this requirement cannot be provided
Problem: the to be installed patch:openSUSE-SLE-15.4-2022-4262-1.noarch conflicts with 'libdevmapper1_03-32bit.x86_64 < 2.03.05_1.02.163-150400.185.1' provided by the installed libdevmapper1_03-32bit-1.02.163-150400.178.1.x86_64
Problem: the to be installed patch:openSUSE-SLE-15.4-2023-161-1.noarch conflicts with 'python-py.noarch < 1.10.0-150100.5.12.1' provided by the installed python-py-1.8.1-11.12.4.noarch
Solution 1: deinstallation of python-py-1.8.1-11.12.4.noarch
Solution 2: do not install patch:openSUSE-SLE-15.4-2023-161-1.noarch
Choose from above solutions by number or skip, retry or cancel [1/2/s/r/c/d/?] (c): c
so I manually logged in and selected solutions for the problem by deinstalling some python packages and concluded the patching.
On baremetal-support the openQA package wouldn't upgrade due to conflicting user IDs regarding geekotest as there is an NFS mount used.
So I changed user ids and updated ownership:
systemctl stop postgresql openqa-webui openqa-scheduler
usermod -u 17256 rbrown
usermod -u 1001 geekotest
umount /var/lib/openqa/share/factory/iso
chown -R geekotest /var/lib/openqa/
find / -uid 483 -exec chown -v -h 1001 '{}' \;
zypper dup
systemctl start default
mount -a
Updated by okurz over 1 year ago
- Subject changed from jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time to jenkins package (and others) not upgraded on jenkins.qa.suse.de since some time size:M
- Description updated (diff)
Updated by okurz over 1 year ago
- Due date deleted (
2023-05-06) - Status changed from In Progress to Resolved
On openqa-piworker I changed priorities for repositories and upgraded, some more packages from devel:python:backports and now it looks good.
I merged https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/846 . It was deployed cleanly and salt state looks good as well. All generic hosts including jenkins.qa.suse.de are now up-to-date and should be kept as such.