action #135134
closed[tools] GitlabCI salt-pillars-openqa deploy failed on baremetal-support.qa.suse.de
0%
Description
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1803184
baremetal-support.qa.suse.de:
Data failed to compile:
----------
Rendering SLS 'base:debug_poo133469' failed: mapping values are not allowed here; line 13
---
[...]
attempts: 5
minion_cmd_file:
file.patch:
- name: warning: waiting for shared lock on /usr/lib/sysimage/rpm/Packages <======================
error: cannot get shared lock on /usr/lib/sysimage/rpm/Packages
error: cannot open Packages index using db4 - Operation not permitted (1)
error: cannot open Packages database in /usr/lib/sysimage/rpm
warning: waiting for shared lock on /usr/lib/sysimage/rpm/Packages
error: cannot get shared lock on /usr/lib/sysimage/rpm/Packages
[...]
---
section_end:1693819642:step_script
~~~
Updated by okurz 8 months ago
- Related to action #133469: [alert] Salt states don't apply sometimes on individual workers size:M added
Updated by nicksinger 8 months ago
- Status changed from New to In Progress
- Assignee set to nicksinger
I will look into this. Looks a little bit like the command to find the correct minion.py file via an rpm query failed
Updated by nicksinger 8 months ago
The missing quotes actually saved us as rpm was/is stuck on the system. Looking at ps I see that apparently the last auto-upgrade never finished due to disk wait:
root 17634 0.0 0.0 8736 2136 ? Ss Sep02 0:00 /bin/sh -c zypper -n --non-interactive-include-reboot-patches dup --replacefiles --auto-agree-with-licenses --download-in-advance && needs-restarting --reboothint >/dev/null || (command -v rebootmgrctl >/dev/null && rebootmgrctl reboot ||:)
root 17639 0.0 4.5 592088 173416 ? Sl Sep02 0:25 \_ zypper -n --non-interactive-include-reboot-patches dup --replacefiles --auto-agree-with-licenses --download-in-advance
root 17876 0.0 0.1 48404 5584 ? S Sep02 0:00 \_ /usr/bin/systemd-inhibit --what=sleep:shutdown:idle --who=zypp --mode=block --why=Zypp commit running. /usr/bin/cat
root 17877 0.0 0.0 3040 0 ? S Sep02 0:00 | \_ /usr/bin/cat
root 17878 0.0 0.0 8736 2248 ? S Sep02 0:00 \_ /bin/bash /usr/lib/zypp/plugins/commit/btrfs-defrag-plugin.sh
root 17884 0.0 0.1 42852 5044 ? S Sep02 0:00 \_ /usr/lib/zypp/plugins/commit/snapper-zypp-plugin
root 17885 0.0 0.1 19616 6336 ? S Sep02 0:00 \_ /usr/bin/python3 /usr/lib/zypp/plugins/commit/zyppnotify
root 18027 0.0 0.9 63136 34824 ? D Sep02 0:00 \_ rpm --root / --dbpath /usr/lib/sysimage/rpm -U --percent --noglob --force --nodeps -- /var/cache/zypp/packages/devel_openQA/x86_64/openQA-4.6.1693565748.f367431-lp154.6040.1.x86_64.rpm
Updated by nicksinger 8 months ago
- Status changed from In Progress to Resolved
nfs on /var/lib/openqa/share/factory/iso hang and I had to umount -l it. Not sure why this made rpm from auto-upgrade hang. Also had to kill this hanging rpm command with kill -9 afterwards. This freed the lock and I was able to query the rpm database again. With that the pipeline ran again: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1803361
We decided to not look further into the issue and see if it comes up again before investing more time.