action #12616
closedautomatic needle committing from openQA on osd fails silently
0%
Description
observation¶
needles from /var/lib/openqa/share/tests/sle/products/sle/needles on osd are not auto-commited and there is no error message displayed in the web ui as we had in before.
steps to reproduce¶
- create new needle on webUI
- observe error messages in /var/log/openqa
problem¶
- There is no error reporting anymore in webUI
- openQA processes take too much memory on osd which make the git call fail (OOM)
suggestion¶
- Add back better error reporting
- Add workaround to restart openQA webUI every night
- Fix memory leak/overuse in the long run
workaround¶
As geekotest on osd manually check the git repo and commit and push if necessary.
Updated by okurz over 8 years ago
[06/07/2016 10:38:33] okurz: reading the documentation of hypnotoad (and trying on production whistle) - you can kill -QUIT the workers
[06/07/2016 10:38:42] okurz: then they will stop gracefully and get restarted
[06/07/2016 10:39:34] http://mojolicious.org/perldoc/Mojo/Server/Hypnotoad#MANAGER-SIGNALS
Updated by coolo over 8 years ago
The magic command line to free memory:
pkill -QUIT -f 'openqa prefork' -P $(pgrep -f 'openqa prefork' -P 1)
Updated by coolo over 8 years ago
- Status changed from New to Resolved
this hasn't been seen since I added the cronjobs.
Updated by okurz over 8 years ago
Symptoms seen again although could be different root cause:
#12128#note-1