Project

General

Profile

action #80106

openQA Project - coordination #39719: [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier

openQA Project - coordination #62420: [epic] Distinguish all types of incompletes

corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily

Added by okurz 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
nicksinger
Target version:
openQA Project - Ready
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

To find out if worker cache services corrupt the sqlite database due to being killed on systemd service termination we should enlarge the timeout on o3 and osd of all relevant worker systemd services temporarily, e.g. 600s, and monitor the situation

Suggestions

On all o3+osd worker machines override the default default 90s kill timeout of services openqa-worker@* openqa-worker-cacheservice.service openqa-worker-cacheservice-minion.service

Related issues

Copied from openQA Project - action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retryResolved2020-05-18

Copied to openQA Infrastructure - action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cacheResolved2020-11-26

History

#1 Updated by okurz 2 months ago

  • Copied from action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added

#2 Updated by okurz 2 months ago

  • Subject changed from Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*database disk image is malformed.*":retry to corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily

#3 Updated by nicksinger 2 months ago

  • Status changed from New to In Progress

#4 Updated by nicksinger 2 months ago

I checked with systemctl list-unit-files | grep openqa on openqaworker9 the relevant .service files. From what I can see there I picked the following to edit their timeout:

which are actually exactly the three Oli already mentioned :)
I then issued the following on OSD to apply it on all OSD workers:
salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload'.
To ensure everything is in place and correctly loaded I checked the output of salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do systemctl cat $service | grep 300; done' and every worker shows 3 lines of output -> everything worked as expected.

Next is o3 where I don't have salt :(

#5 Updated by nicksinger 2 months ago

o3 hosts aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester and rebel got changed with for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload && ls /etc/systemd/system/*.d/. All machines have the expected files. I'm just unsure if it will survive reboots :)

#6 Updated by nicksinger 2 months ago

  • Status changed from In Progress to Feedback

#7 Updated by okurz about 2 months ago

nicksinger wrote:

All machines have the expected files. I'm just unsure if it will survive reboots :)

I checked. It did :)

#8 Updated by nicksinger about 2 months ago

  • Status changed from Feedback to Resolved

#9 Updated by okurz about 2 months ago

  • Copied to action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache added

Also available in: Atom PDF