action #80106
closed
openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues
openQA Project - coordination #62420: [epic] Distinguish all types of incompletes
corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
Added by okurz over 3 years ago.
Updated over 3 years ago.
Description
Motivation¶
To find out if worker cache services corrupt the sqlite database due to being killed on systemd service termination we should enlarge the timeout on o3 and osd of all relevant worker systemd services temporarily, e.g. 600s, and monitor the situation
Suggestions¶
On all o3+osd worker machines override the default default 90s kill timeout of services openqa-worker@* openqa-worker-cacheservice.service openqa-worker-cacheservice-minion.service
- Copied from action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added
- Subject changed from Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*database disk image is malformed.*":retry to corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
- Status changed from New to In Progress
I checked with systemctl list-unit-files | grep openqa on openqaworker9 the relevant .service files. From what I can see there I picked the following to edit their timeout:
which are actually exactly the three Oli already mentioned :)
I then issued the following on OSD to apply it on all OSD workers:
salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload'.
To ensure everything is in place and correctly loaded I checked the output of salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do systemctl cat $service | grep 300; done' and every worker shows 3 lines of output -> everything worked as expected.
Next is o3 where I don't have salt :(
o3 hosts aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester and rebel got changed with for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload && ls /etc/systemd/system/*.d/. All machines have the expected files. I'm just unsure if it will survive reboots :)
- Status changed from In Progress to Feedback
nicksinger wrote:
All machines have the expected files. I'm just unsure if it will survive reboots :)
I checked. It did :)
- Status changed from Feedback to Resolved
- Copied to action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache added
Also available in: Atom
PDF