QA » openQA Project

That sounds even better.

Adding Wants=time-sync.target to openqa-worker.target should make this even easier. We don't need to write code to handle this as long as whatever service is being used includes time-set.target in Before= and Wants=. And if nothing implements it it'll be racy like it is now:

https://github.com/os-autoinst/openQA/pull/4692

I did some tests using the new packages with the updated unit file, but it doesn't seem to do the trick:

openqa-piworker:~ # systemctl cat openqa-worker@3
# /usr/lib/systemd/system/openqa-worker-plain@.service
# unit description file for openQA worker processes
# start using e.g.
# systemctl start openqa-worker@1.service
# replace '1' with the instance number you want
[Unit]
Description=openQA Worker #%i
After=openqa-slirpvde.service network.target nss-lookup.target remote-fs.target time-sync.target
PartOf=openqa-worker.target

[Service]
Type=simple
PermissionsStartOnly=True
ExecStartPre=/usr/bin/install -d -m 0755 -o _openqa-worker /var/lib/openqa/pool/%i
ExecStart=/usr/share/openqa/script/worker --instance %i
User=_openqa-worker
KillMode=mixed
Restart=on-failure
Delegate=yes
Slice=openqa-worker.slice

[Install]
WantedBy=multi-user.target

openqa-piworker:~ # systemctl status openqa-worker@3
● openqa-worker-plain@3.service - openQA Worker #3
     Loaded: loaded (/usr/lib/systemd/system/openqa-worker-plain@.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Tue 2022-04-19 12:00:41 UTC; 1 month 21 days ago
    Process: 1705 ExecStartPre=/usr/bin/install -d -m 0755 -o _openqa-worker /var/lib/openqa/pool/3 (code=exited, status=0/SUCCESS)
    Process: 1751 ExecStart=/usr/share/openqa/script/worker --instance 3 (code=exited, status=0/SUCCESS)
   Main PID: 1751 (code=exited, status=0/SUCCESS)

Apr 19 12:00:41 openqa-piworker worker[1751]:  - websocket API version: 1
Apr 19 12:00:41 openqa-piworker worker[1751]:  - web UI hosts:          openqa.suse.de
Apr 19 12:00:41 openqa-piworker worker[1751]:  - class:                 generalhw_RPi3B
Apr 19 12:00:41 openqa-piworker worker[1751]:  - no cleanup:            no
Apr 19 12:00:41 openqa-piworker worker[1751]:  - pool directory:        /var/lib/openqa/pool/3
Apr 19 12:00:41 openqa-piworker worker[1751]: [info] Project dir for host openqa.suse.de is /var/lib/openqa/share
Apr 19 12:00:41 openqa-piworker worker[1751]: [info] Registering with openQA openqa.suse.de
Apr 19 12:00:41 openqa-piworker worker[1751]: [error] Failed to register at openqa.suse.de - 403 response: timestamp mismatch - ignoring server
Apr 19 12:00:41 openqa-piworker worker[1751]: [error] Stopping because registration with all configured web UI hosts failed
Apr 19 12:00:41 openqa-piworker systemd[1]: openqa-worker-plain@3.service: Succeeded.

What really seems to work(around the issue) is:

# /etc/systemd/system/openqa-worker-plain@.service.d/override.conf
[Service]
Restart=always
RestartSec=10s

Note that Restart=on-failure won't work as openQA-worker returns 0 even in an error condition (which might make sense to be changed as well).

Sorry about the delayed reply. I think Redmine even tried to remind me but the emails with custom queries were not very clear and I don't see future tickets in due date-queries.

dheidler wrote:

I did some tests using the new packages with the updated unit file, but it doesn't seem to do the trick:

Can you confirm that cronyd was started before?

Note that Restart=on-failure won't work as openQA-worker returns 0 even in an error condition (which might make sense to be changed as well).

I filed #113240.

cdywan wrote:

Marius suggests as another option we could catch the timestamp mismatch and retry in this specific case so I will look into that.

I'm approaching this by re-using the code to retry like in other cases where registration failed but isn't a fatal error, and OPENQA_WORKER_CONNECT_INTERVAL is respected here as well: https://github.com/os-autoinst/openQA/pull/4751

Btw another potential work-around I saw while looking into that code: Authentication via personal access tokens instead of API keys is not affected by the timestamp issue.