Project

General

Profile

action #73342

all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry

Added by okurz 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2020-10-14
Due date:
2020-10-16
% Done:

0%

Estimated time:

Description

jobs run on openqaworker8 incomplete, the reason is Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed at /usr/share/openqa/script/../lib/OpenQA/CacheService/Model/Downloads.pm line 34. at /usr/share/openqa/script/../lib/OpenQA/CacheSer…

Please see more details on:
https://10.160.0.207/tests/4823712
https://10.160.0.207/tests/4821829


Related issues

Related to openQA Project - action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retryResolved2020-05-18

Related to openQA Infrastructure - action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8Resolved2020-11-162020-11-18

Copied from openQA Project - action #73321: all jobs run on openqaworker8 incomplete:"Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed*"Rejected2020-10-14

Copied to openQA Infrastructure - action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retryResolved

History

#1 Updated by okurz 3 months ago

  • Copied from action #73321: all jobs run on openqaworker8 incomplete:"Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed*" added

#2 Updated by okurz 3 months ago

copied the sqlite database file so that one can investigate further, see #73321 for this. To stop any further problematic jobs produced by this worker I did:

systemctl stop openqa-worker* salt-minion telegraf

I see that the worker was not rebooted for longer but I expected that to have happened automatically. Will check package versions and reboot which should clean out the pool and cache automatically anyway and fix the reported issue.

EDIT: Found out that many workers have not automatically rebooted since 31 days. Manually triggering the "auto-update" service on openqaworker8 and taking a look into the corresponding journal shows that there is an "interactive" patch pending which seems to be a kernel upgrade. So trying with the flag "--non-interactive-include-reboot-patches" now: systemctl daemon-reload; systemctl start auto-update ; journalctl -f -u auto-update. This triggered the kernel upgrade and asked the rebootmgr to reboot during the next maintenance window. For the above reasons I triggered a reboot now.

#3 Updated by okurz 3 months ago

  • Due date set to 2020-10-16
  • Status changed from In Progress to Feedback

Machine is rebooted, has a clean cache db and is working on jobs fine for now. Will monitor over the next time.

#4 Updated by okurz 3 months ago

  • Related to action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added

#5 Updated by okurz 3 months ago

  • Subject changed from all jobs run on openqaworker8 incomplete:"Cache service status error from API: Minion job #46203 failed: Couldn't add download: DBD::SQLite::st execute failed: database disk image is malformed*" to all jobs run on openqaworker8 incomplete: auto_review:"Cache service status error from API: Minion job .*failed: .*database disk image is malformed*":retry

#6 Updated by okurz 3 months ago

  • Subject changed from all jobs run on openqaworker8 incomplete: auto_review:"Cache service status error from API: Minion job .*failed: .*database disk image is malformed*":retry to all jobs run on openqaworker8 incomplete: auto_review:"Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry

#7 Updated by okurz 3 months ago

  • Priority changed from Urgent to Normal

I monitored all openQA worker instances since the reboot and when the cache database was re-initialized and so far I have seen a lot of passed jobs on all worker instances. So it seems the problem of corrupt cache database was resolved as well as the problem that the machine(s) did not apply "interactive" patches: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/380

EDIT: Seems like the alerts for "incompletes" are also related to this so I paused the alerts for https://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&panelId=17&fullscreen&edit&tab=alert and https://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&panelId=14&fullscreen&edit&tab=alert for now until the situation is repaired.

#8 Updated by okurz 3 months ago

  • Status changed from Feedback to Resolved

merged, alerts reactivated, done.

#9 Updated by okurz 3 months ago

  • Subject changed from all jobs run on openqaworker8 incomplete: auto_review:"Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry to all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry

#10 Updated by okurz 3 months ago

  • Copied to action #75220: all jobs run on openqaworker8 incomplete: "Cache service status error from API: Minion job .*failed: .*(database disk image is malformed|not a database)":retry added

#11 Updated by okurz 2 months ago

  • Related to action #78058: [Alerting] Incomplete jobs of last 24h alert - again many incompletes due to corrupted cache, on openqaworker8 added

Also available in: Atom PDF