action #80106
openQA Project - coordination #39719: [saga][epic] Detect "known failures" and mark jobs as such to make tests more stable, reviewing test results and tracking known issues easier
openQA Project - coordination #62420: [epic] Distinguish all types of incompletes
corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
0%
Description
Motivation¶
To find out if worker cache services corrupt the sqlite database due to being killed on systemd service termination we should enlarge the timeout on o3 and osd of all relevant worker systemd services temporarily, e.g. 600s, and monitor the situation
Suggestions¶
On all o3+osd worker machines override the default default 90s kill timeout of services openqa-worker@* openqa-worker-cacheservice.service openqa-worker-cacheservice-minion.service
Related issues
History
#1
Updated by okurz 2 months ago
- Copied from action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added
#3
Updated by nicksinger 2 months ago
- Status changed from New to In Progress
#4
Updated by nicksinger 2 months ago
I checked with systemctl list-unit-files | grep openqa
on openqaworker9 the relevant .service files. From what I can see there I picked the following to edit their timeout:
- openqa-worker-cacheservice-minion.service
- openqa-worker-cacheservice.service
- openqa-worker@.service
which are actually exactly the three Oli already mentioned :)
I then issued the following on OSD to apply it on all OSD workers:
salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload'
.
To ensure everything is in place and correctly loaded I checked the output of salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do systemctl cat $service | grep 300; done'
and every worker shows 3 lines of output -> everything worked as expected.
Next is o3 where I don't have salt :(
#5
Updated by nicksinger 2 months ago
o3 hosts aarch64
openqaworker1
openqaworker4
openqaworker7
power8
imagetester
and rebel
got changed with for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload && ls /etc/systemd/system/*.d/
. All machines have the expected files. I'm just unsure if it will survive reboots :)
#6
Updated by nicksinger 2 months ago
- Status changed from In Progress to Feedback
#7
Updated by okurz about 2 months ago
nicksinger wrote:
All machines have the expected files. I'm just unsure if it will survive reboots :)
I checked. It did :)
#8
Updated by nicksinger about 2 months ago
- Status changed from Feedback to Resolved
#9
Updated by okurz about 2 months ago
- Copied to action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache added