action #68164
Updated by okurz almost 4 years ago
## Observation
reported in irc room [#opensuse-factory](irc://chat.freenode.net/opensuse-factory) :
```
[17/06/2020 09:02:45] <Dimstar> Good morning all; anybody knows what's up with o3 not picking up the scheduled jobs?
[17/06/2020 09:04:44] <guillaume_g> Dimstar: Hi! :) Workers are reported broken
[17/06/2020 09:05:00] <guillaume_g> Dimstar: "No workers active in the cache service"
[17/06/2020 09:05:30] <guillaume_g> the only workers which are running are the one which are not auto-updated ;)
[17/06/2020 09:07:31] <guillaume_g> kraih: ^ could it be your MR https://github.com/os-autoinst/openQA/pull/3177 ?
[17/06/2020 09:07:32] <|Anna|> Github project os-autoinst/openQA pull request#3177: "Reset locks when restarting the cache service Minion worker", created on 2020-06-16, status: closed on 2020-06-16, https://github.com/os-autoinst/openQA/pull/3177
[17/06/2020 09:08:09] <fvogt> At least on openqa-aarch64 all services are up and running, according to systemctl
[17/06/2020 09:09:40] <guillaume_g> Dimstar: Could you abort openSUSE:Factory:ARM:Live/JeOS:GNOME-efi.aarch64 please?
[17/06/2020 09:09:50] <Dimstar> fun - worker info for e.g. ow1:1 is alive, last seen less than a minute ago, broken
[17/06/2020 09:10:06] <Dimstar> guillaume_g: done
[17/06/2020 09:12:02] <fvogt> openqa-worker-cacheservice-minion.service is dead - it printed usage info...
[17/06/2020 09:12:09] <fvogt> " See 'APPLICATION help COMMAND' for more information on a specific command."
[17/06/2020 09:12:24] <fvogt> For some reason that has exit code 0, which isn't helpful
[17/06/2020 09:13:49] <guillaume_g> Dimstar: thanks! :)
[17/06/2020 09:14:07] <fvogt> It's the order of arguments
[17/06/2020 09:14:15] <fvogt> It has to be "run -m production", not "-m production run"
[17/06/2020 09:16:41] <fvogt> Started it manually, worker is back. So confirmed to be that indeed
[17/06/2020 09:23:58] <guillaume_g> Great!
[17/06/2020 09:24:22] <fvogt> Now we just need someone to commit and push the fix
[17/06/2020 09:24:53] <Dimstar> fvogt: did you restart all workers for this? e.g. ow1, ow4 ow7, imagetester?
[17/06/2020 09:25:19] <fvogt> Dimstar: Where happened to your 'S'?
[17/06/2020 09:25:30] <fvogt> No, I only tried to prove the theory on openqa-aarch64
[17/06/2020 09:25:58] <fvogt> You can run su _openqa-worker -c '/usr/share/openqa/script/openqa-workercache run -m production --reset-locks' if you want to
[17/06/2020 09:26:17] <Dimstar> fvogt: ok; that's fine; just needed to know... I'll kick the x86_64 workers
```
## Lessons learned + TODOs
* Ask explicitly how changes to systemd files have been tested
* Add tests for systemd services and/or the daemon wrapper scripts