Project

General

Profile

Actions

action #80106

closed

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

openQA Project - coordination #62420: [epic] Distinguish all types of incompletes

corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily

Added by okurz over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

To find out if worker cache services corrupt the sqlite database due to being killed on systemd service termination we should enlarge the timeout on o3 and osd of all relevant worker systemd services temporarily, e.g. 600s, and monitor the situation

Suggestions

On all o3+osd worker machines override the default default 90s kill timeout of services openqa-worker@* openqa-worker-cacheservice.service openqa-worker-cacheservice-minion.service


Related issues 2 (0 open2 closed)

Copied from openQA Project - action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retryResolvedmkittler2020-05-18

Actions
Copied to openQA Infrastructure - action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cacheResolvednicksinger2020-11-26

Actions
Actions #1

Updated by okurz over 3 years ago

  • Copied from action #67000: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry added
Actions #2

Updated by okurz over 3 years ago

  • Subject changed from Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*database disk image is malformed.*":retry to corrupted worker cache sqlite: Enlarge systemd service kill timeout temporarily
Actions #3

Updated by nicksinger over 3 years ago

  • Status changed from New to In Progress
Actions #4

Updated by nicksinger over 3 years ago

I checked with systemctl list-unit-files | grep openqa on openqaworker9 the relevant .service files. From what I can see there I picked the following to edit their timeout:

which are actually exactly the three Oli already mentioned :)
I then issued the following on OSD to apply it on all OSD workers:
salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload'.
To ensure everything is in place and correctly loaded I checked the output of salt -l error -C 'G@roles:worker' cmd.run 'for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do systemctl cat $service | grep 300; done' and every worker shows 3 lines of output -> everything worked as expected.

Next is o3 where I don't have salt :(

Actions #5

Updated by nicksinger over 3 years ago

o3 hosts aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester and rebel got changed with for service in openqa-worker-cacheservice-minion.service openqa-worker-cacheservice.service openqa-worker@.service; do mkdir -p "/etc/systemd/system/$service.d" && echo -e "[Service]\nTimeoutStopSec=300" > "/etc/systemd/system/$service.d/poo80106.conf"; done && systemctl daemon-reload && ls /etc/systemd/system/*.d/. All machines have the expected files. I'm just unsure if it will survive reboots :)

Actions #6

Updated by nicksinger over 3 years ago

  • Status changed from In Progress to Feedback
Actions #7

Updated by okurz over 3 years ago

nicksinger wrote:

All machines have the expected files. I'm just unsure if it will survive reboots :)

I checked. It did :)

Actions #8

Updated by nicksinger over 3 years ago

  • Status changed from Feedback to Resolved
Actions #9

Updated by okurz over 3 years ago

  • Copied to action #80408: revert longer timeout override for openQA services as we could not see less problems with corrupted worker cache added
Actions

Also available in: Atom PDF