action #135134
closed[tools] GitlabCI salt-pillars-openqa deploy failed on baremetal-support.qa.suse.de
0%
Description
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1803184
baremetal-support.qa.suse.de:
Data failed to compile:
----------
Rendering SLS 'base:debug_poo133469' failed: mapping values are not allowed here; line 13
---
[...]
attempts: 5
minion_cmd_file:
file.patch:
- name: warning: waiting for shared lock on /usr/lib/sysimage/rpm/Packages <======================
error: cannot get shared lock on /usr/lib/sysimage/rpm/Packages
error: cannot open Packages index using db4 - Operation not permitted (1)
error: cannot open Packages database in /usr/lib/sysimage/rpm
warning: waiting for shared lock on /usr/lib/sysimage/rpm/Packages
error: cannot get shared lock on /usr/lib/sysimage/rpm/Packages
[...]
---
section_end:1693819642:step_script
~~~
Updated by okurz 11 months ago
- Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
Updated by nicksinger 11 months ago
- Status changed from New to In Progress
- Assignee set to nicksinger
I will look into this. Looks a little bit like the command to find the correct minion.py file via an rpm query failed
Updated by nicksinger 11 months ago
The missing quotes actually saved us as rpm was/is stuck on the system. Looking at ps I see that apparently the last auto-upgrade never finished due to disk wait:
root 17634 0.0 0.0 8736 2136 ? Ss Sep02 0:00 /bin/sh -c zypper -n --non-interactive-include-reboot-patches dup --replacefiles --auto-agree-with-licenses --download-in-advance && needs-restarting --reboothint >/dev/null || (command -v rebootmgrctl >/dev/null && rebootmgrctl reboot ||:)
root 17639 0.0 4.5 592088 173416 ? Sl Sep02 0:25 \_ zypper -n --non-interactive-include-reboot-patches dup --replacefiles --auto-agree-with-licenses --download-in-advance
root 17876 0.0 0.1 48404 5584 ? S Sep02 0:00 \_ /usr/bin/systemd-inhibit --what=sleep:shutdown:idle --who=zypp --mode=block --why=Zypp commit running. /usr/bin/cat
root 17877 0.0 0.0 3040 0 ? S Sep02 0:00 | \_ /usr/bin/cat
root 17878 0.0 0.0 8736 2248 ? S Sep02 0:00 \_ /bin/bash /usr/lib/zypp/plugins/commit/btrfs-defrag-plugin.sh
root 17884 0.0 0.1 42852 5044 ? S Sep02 0:00 \_ /usr/lib/zypp/plugins/commit/snapper-zypp-plugin
root 17885 0.0 0.1 19616 6336 ? S Sep02 0:00 \_ /usr/bin/python3 /usr/lib/zypp/plugins/commit/zyppnotify
root 18027 0.0 0.9 63136 34824 ? D Sep02 0:00 \_ rpm --root / --dbpath /usr/lib/sysimage/rpm -U --percent --noglob --force --nodeps -- /var/cache/zypp/packages/devel_openQA/x86_64/openQA-4.6.1693565748.f367431-lp154.6040.1.x86_64.rpm
Updated by nicksinger 11 months ago
- Status changed from In Progress to Resolved
nfs on /var/lib/openqa/share/factory/iso
hang and I had to umount -l
it. Not sure why this made rpm from auto-upgrade hang. Also had to kill this hanging rpm command with kill -9
afterwards. This freed the lock and I was able to query the rpm database again. With that the pipeline ran again: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1803361
We decided to not look further into the issue and see if it comes up again before investing more time.