action #80106
closedopenQA Project (public) - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues
openQA Project (public) - coordination #62420: [epic] Distinguish all types of incompletes
corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
0%
Description
Motivation¶
To find out if worker cache services corrupt the sqlite database due to being killed on systemd service termination we should enlarge the timeout on o3 and osd of all relevant worker systemd services temporarily, e.g. 600s, and monitor the situation
Suggestions¶
On all o3+osd worker machines override the default default 90s kill timeout of services openqa-worker@* openqa-worker-cacheservice.service openqa-worker-cacheservice-minion.service
Updated by okurz about 4 years ago
- Copied from action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added
Updated by okurz about 4 years ago
- Subject changed from Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*database disk image is malformed.*":retry to corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
Updated by nicksinger about 4 years ago
I checked with systemctl list-unit-files | grep openqa on openqaworker9 the relevant .service files. From what I can see there I picked the following to edit their timeout:
- openqa-worker-cacheservice-minion.service
- openqa-worker-cacheservice.service
- openqa-worker@.service
which are actually exactly the three Oli already mentioned :)
I then issued the following on OSD to apply it on all OSD workers:
salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload'.
To ensure everything is in place and correctly loaded I checked the output of salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do systemctl cat $service | grep 300; done' and every worker shows 3 lines of output -> everything worked as expected.
Next is o3 where I don't have salt :(
Updated by nicksinger about 4 years ago
o3 hosts aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester and rebel got changed with for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload && ls /etc/systemd/system/*.d/. All machines have the expected files. I'm just unsure if it will survive reboots :)
Updated by nicksinger about 4 years ago
- Status changed from In Progress to Feedback
Updated by okurz about 4 years ago
nicksinger wrote:
All machines have the expected files. I'm just unsure if it will survive reboots :)
I checked. It did :)
Updated by nicksinger about 4 years ago
- Status changed from Feedback to Resolved
Updated by okurz about 4 years ago
- Copied to action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache added