openQA Project (public) » openQA Infrastructure (public)

Today OSD's deployment failed because the rpm database on arm-1 is broken again:

    Problem occurred during or after installation or removal of packages:
    Failed to cache rpm database (1).
    History:
     - 'rpmdb2solv' '-r' '/' '-D' '/usr/lib/sysimage/rpm' '-X' '-p' '/etc/products.d' '/var/cache/zypp/solv/@System/solv' '-o' '/var/cache/zypp/solv/@System/solvqST4AK'
       rpmdb2solv: inconsistent rpm database, key 2106 not found. run 'rpm --rebuilddb' to fix.

    Please see the above error message for a hint.
ERROR: Minions returned with non-zero exit code

(from https://gitlab.suse.de/openqa/osd-deployment/-/jobs/870107)

We actually try to rebuild the rpm database automatically in when we spot the corresponding error message and it has indeed be logged:

2022-03-07 11:15:41 <2> openqaworker-arm-1(13721) [zypp] TargetImpl.cc(buildCache):1068   rpmdb2solv: inconsistent rpm database, key 2106 not found. run 'rpm --rebuilddb' to fix.
2022-03-07 11:15:41 <2> openqaworker-arm-1(13721) [zypp::exec] abstractspawnengine.cc(checkStatus):176 Pid 27052 exited with status 1
2022-03-07 11:15:41 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186 TargetImpl.cc(buildCache):1081 THROW:    Failed to cache rpm database (1).
…
2022-03-07 11:15:42 <1> openqaworker-arm-1(13721) [zypp::plugin++] PluginScript.cc(close):251 PluginScript[13786] /usr/lib/zypp/plugins/commit/zyppnotify -> [0] Disconnect
2022-03-07 11:15:42 <2> openqaworker-arm-1(13721) [zypp::exec] abstractspawnengine.cc(checkStatus):197 Pid 13777 was killed by signal 9 (Killed; Out of memory?)
2022-03-07 11:15:42 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186 solve-commit.cc(solve_and_commit):950 CAUGHT:   Failed to cache rpm database (1).
2022-03-07 11:15:42 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186 History:
2022-03-07 11:15:42 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186  - 'rpmdb2solv' '-r' '/' '-D' '/usr/lib/sysimage/rpm' '-X' '-p' '/etc/products.d' '/var/cache/zypp/solv/@System/solv' '-o' '/var/cache/zypp/solv/@System/solvqST4AK'
2022-03-07 11:15:42 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186    rpmdb2solv: inconsistent rpm database, key 2106 not found. run 'rpm --rebuilddb' to fix.
2022-03-07 11:15:42 <5> openqaworker-arm-1(13721) [zypp-core] Exception.cc(log):186
2022-03-07 11:15:42 <2> openqaworker-arm-1(13721) [zypper] Zypper.h(setExitCode):162 setExitCode 8
2022-03-07 11:15:42 <2> openqaworker-arm-1(13721) [zypper] Zypper.h(setExitCode):162 setExitCode 8
2022-03-07 11:15:42 <1> openqaworker-arm-1(13721) [zypper] Zypper.cc(doCommand):677 Done
2022-03-07 11:15:42 <1> openqaworker-arm-1(13721) [zypper] Zypper.cc(cleanup):729 START
2022-03-07 11:15:42 <1> openqaworker-arm-1(13721) [zypper] main.cc(~Bye):98 ===== Exiting main(8) =====

I assume the problem is that the log messages only occurred within the zypper call made in the deployment step but we only check whether such errors were logged before. (There are no earlier messages in the log than the one mentioned above.)