Project

General

Profile

Actions

action #12616

closed

automatic needle committing from openQA on osd fails silently

Added by okurz almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
2016-07-06
Due date:
% Done:

0%

Estimated time:

Description

observation

needles from /var/lib/openqa/share/tests/sle/products/sle/needles on osd are not auto-commited and there is no error message displayed in the web ui as we had in before.

steps to reproduce

  • create new needle on webUI
  • observe error messages in /var/log/openqa

problem

  1. There is no error reporting anymore in webUI
  2. openQA processes take too much memory on osd which make the git call fail (OOM)

suggestion

  • Add back better error reporting
  • Add workaround to restart openQA webUI every night
  • Fix memory leak/overuse in the long run

workaround

As geekotest on osd manually check the git repo and commit and push if necessary.

Actions #1

Updated by okurz almost 8 years ago

[06/07/2016 10:38:33] okurz: reading the documentation of hypnotoad (and trying on production whistle) - you can kill -QUIT the workers
[06/07/2016 10:38:42] okurz: then they will stop gracefully and get restarted
[06/07/2016 10:39:34] http://mojolicious.org/perldoc/Mojo/Server/Hypnotoad#MANAGER-SIGNALS

Actions #2

Updated by coolo over 7 years ago

The magic command line to free memory:
pkill -QUIT -f 'openqa prefork' -P $(pgrep -f 'openqa prefork' -P 1)

Actions #3

Updated by coolo over 7 years ago

  • Status changed from New to Resolved

this hasn't been seen since I added the cronjobs.

Actions #4

Updated by okurz over 7 years ago

Symptoms seen again although could be different root cause:
#12128#note-1

Actions

Also available in: Atom PDF