action #180701
closedPackage cache on o3 workers does not include any os-autoinst packages size:S
0%
Description
Observation¶
Trying to do a rollback for #180695 I found that there are no cached versions of os-autoinst and mostly only one package which is useless:
okurz@ariel:~> hosts="openqaworker20 openqaworker21 openqaworker22 openqaworker23 openqaworker24 openqaworker25 openqaworker26 openqaworker27 openqaworker28 openqaworker-arm21 openqaworker-arm22 qa-power8-3"; for i in $hosts; do echo "### $i" && ssh root@$i "find /var/cache/zypp/packages/" ; done
### openqaworker20
/var/cache/zypp/packages/
/var/cache/zypp/packages/repo-backports-update
/var/cache/zypp/packages/repo-backports-update/x86_64
/var/cache/zypp/packages/repo-backports-update/noarch
/var/cache/zypp/packages/repo-sle-update
/var/cache/zypp/packages/repo-sle-update/noarch
/var/cache/zypp/packages/repo-sle-update/x86_64
/var/cache/zypp/packages/devel_openQA_Leap
/var/cache/zypp/packages/devel_openQA_Leap/noarch
/var/cache/zypp/packages/devel_openQA_Leap/x86_64
/var/cache/zypp/packages/devel_openQA
/var/cache/zypp/packages/devel_openQA/noarch
/var/cache/zypp/packages/devel_openQA/x86_64
/var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-4.6.1715785524.aacf9bc3-lp155.6658.1.x86_64.rpm
/var/cache/zypp/packages/repo-oss
/var/cache/zypp/packages/repo-oss/x86_64
/var/cache/zypp/packages/repo-oss/noarch
### openqaworker21
/var/cache/zypp/packages/
/var/cache/zypp/packages/repo-update-oss
/var/cache/zypp/packages/repo-update-oss/noarch
/var/cache/zypp/packages/repo-update-oss/x86_64
/var/cache/zypp/packages/devel_openQA_Leap
/var/cache/zypp/packages/devel_openQA_Leap/x86_64
/var/cache/zypp/packages/devel_openQA_Leap/noarch
/var/cache/zypp/packages/repo-sle-update
/var/cache/zypp/packages/repo-sle-update/noarch
/var/cache/zypp/packages/repo-sle-update/x86_64
/var/cache/zypp/packages/devel_openQA
/var/cache/zypp/packages/devel_openQA/x86_64
/var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-4.6.1715785524.aacf9bc3-lp155.6658.1.x86_64.rpm
/var/cache/zypp/packages/devel_openQA/noarch
/var/cache/zypp/packages/repo-backports-update
/var/cache/zypp/packages/repo-backports-update/x86_64
/var/cache/zypp/packages/repo-backports-update/noarch
/var/cache/zypp/packages/openSUSE-Leap-$releasever-1
/var/cache/zypp/packages/openSUSE-Leap-$releasever-1/x86_64
/var/cache/zypp/packages/openSUSE-Leap-$releasever-1/noarch
…
Possibly a regression from https://github.com/os-autoinst/openQA/commit/d4e1252241d33da29f8228758b95969e7c66cf66
Acceptance criteria¶
- AC1: All o3 workers are ensured to have at least the previously installed version of os-autoinst and all relevant openQA packages
Suggestions¶
- Crosscheck if repos are actually enabled for saving packages. If not then we should consider that in the automatic update scripting or at least adding the according instructions on our o3 setup documentation
- Look into https://github.com/os-autoinst/openQA/commit/d4e1252241d33da29f8228758b95969e7c66cf66
- #174313
- Introduce proper checks to ensure that we don't clean out complete caches again
- Monitor over a reasonable time
- Ensure that all o3 workers have at least the previously installed version of os-autoinst and all relevant openQA packages
Updated by okurz 6 days ago
- Copied from action #180695: tests fail due to removal of '%cmd' from testapi size:S added
Updated by okurz 6 days ago
- Copied to action #180707: o3 worker openqaworker20 has outdated os-autoinst package, possibly inoperative auto-update service size:S added
Updated by gpuliti 6 days ago
- Related to action #174313: [o3][zabbix][alert] / and /var/tmp: "Disk space is low and might be full in 7d (used > 85%)" since 2024-12-11 06:50 size:S added
Updated by gpuliti 6 days ago
- Related to deleted (action #174313: [o3][zabbix][alert] / and /var/tmp: "Disk space is low and might be full in 7d (used > 85%)" since 2024-12-11 06:50 size:S)
Updated by mkittler 6 days ago
The keeppackages=1 is definitely missing on some o3 workers, e.g. /etc/zypp/repos.d/devel_openQA.repo doesn't contain it on openqaworker21. So most likely not a regression from https://github.com/os-autoinst/openQA/commit/d4e1252241d33da29f8228758b95969e7c66cf66 but simply a misconfiguration (or rather the bad default config has never been changed).
Updated by mkittler 5 days ago
- Status changed from In Progress to Resolved
I also mentioned 3 months ago on #174313#note-27 that this is not a regression of my previous change.
So I invoked zypper mr --keep-packages 1 devel_openQA devel_openQA_Leap and mkdir -p /etc/systemd/system/openqa-auto-update.service.d && echo -e \"[Service]\nEnvironment=OPENQA_PACKAGE_CACHE_RETENTION=1\" > /etc/systemd/system/openqa-auto-update.service.d/enable-pkg-cache-retention.conf && systemctl daemon-reload on all o3 workers. I also extended the Wiki.