action #158814
closed
o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm size:M
Added by tinita about 1 month ago.
Updated 27 days ago.
Category:
Regressions/Crashes
Description
Observation¶
From /var/log/openqa:
[2024-04-10T15:20:04.190828Z] [error] [6XDd4QtKL9Pc] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.
[2024-04-10T15:25:06.279066Z] [error] [UYVJthN6XmUF] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.
[2024-04-10T15:45:04.107327Z] [error] [T4fj_bgqmapR] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.
[2024-04-10T15:50:02.488807Z] [error] [xYSu0-F1l8A6] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.
I can't find that in the osd openqa journal.
Looking at the time, it looks related to #158805
Acceptance criteria¶
- AC1: All our services using Minion are restarted automatically as necessary on an update of the perl-Minion package
Suggestions¶
- Catch the above error in active openQA components and trigger a self-restart (without causing an endless loop of restarts)
- Use a systemd path unit to trigger an auto-restart
I just restarted the webui on o3.
It seems it asn't restarted after the perl-Minion update (but still the jquery issue was gone, so I didn't check that it was restarted).
Monitoring the log now..
- Subject changed from o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm to o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm size:M
- Description updated (diff)
- Status changed from New to Workable
- Assignee set to jbaier_cz
If I understood the problem correctly, we want to restart on db migrations, i.e. when /usr/lib/perl5/vendor_perl/5.38.2/Minion/Backend/resources/migrations/pg.sql changes
- Status changed from Workable to In Progress
I installed the changed rpms manually, enable the auto-restart unit via systemctl enable --now openqa-minion-restart.path and after faking Minion update via zypper install --force perl-Minion, I spotted the following:
Apr 11 15:49:33 dagr.qam.suse.cz [RPM][26338]: Transaction ID 6617ea6d started
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Starting Restarts services which are using Minion...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping Handler for live view in openQA's web UI...
Apr 11 15:49:34 dagr.qam.suse.cz openqa-webui-daemon[25830]: [warn] Stopping worker 25984 immediately
...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping The openQA web UI...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping The openQA daemon for various background tasks like cleanup and saving needles...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: openqa-livehandler.service: Deactivated successfully.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopped Handler for live view in openQA's web UI.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Started Handler for live view in openQA's web UI.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: openqa-webui.service: Deactivated successfully.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopped The openQA web UI.
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: install perl-Minion-10.290.0-lp155.2.1.noarch: success
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: install perl-Minion-10.290.0-lp155.2.1.noarch: success
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: Transaction ID 6617ea6d finished: 0
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-gru.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Stopped The openQA daemon for various background tasks like cleanup and saving needles.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Starting Setup local PostgreSQL database for openQA...
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-setup-db.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Finished Setup local PostgreSQL database for openQA.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Started The openQA web UI.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Started The openQA daemon for various background tasks like cleanup and saving needles.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-minion-restart.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Finished Restarts services which are using Minion.
I believe that's what we want.
- Due date set to 2024-04-26
Setting due date based on mean cycle time of SUSE QE Tools
Don't we want it on all openQA instances? If yes then what change should we need in salt states?
okurz wrote in #note-9:
Don't we want it on all openQA instances? If yes then what change should we need in salt states?
The salt change just enables and runs the openqa-minion-restart unit
livdywan wrote in #note-11:
jbaier_cz wrote in #note-8:
The code is merged and deployed. I did systemctl enable --now openqa-minion-restart.path on o3 to enable the service. Do we want that also for osd? If yes: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1155
I guess this isn't quite working yet... or maybe deployed too early?
According to
# ls -l /usr/lib/systemd/system/openqa-minion-restart.path
-rw-r--r-- 1 root root 134 Apr 15 17:46 /usr/lib/systemd/system/openqa-minion-restart.path
that file should be deployed, so it might be a salt thing not understanding I do not want a .service file but a .path unit file
# ls -l /usr/lib/systemd/system/openqa-minion-restart.path
-rw-r--r-- 1 root root 134 Apr 15 17:46 /usr/lib/systemd/system/openqa-minion-restart.path
that file should be deployed, so it might be a salt thing not understanding I do not want a .service file but a .path unit file
Checking the docs, neither salt.states.service nor salt.modules.systemd_service says anything about path units. The implementation seems to recognize it, though?
- Status changed from In Progress to Resolved
Whatever was the reason, it seems it was a temporary issue, second run of the deploy job succeeded. With this, I guess we are done here?
- Status changed from Resolved to Feedback
As discussed in the coord call please pull in the path unit as dependency automatically already in upstream and then remove the special handling in salt again as IIUC that would not be needed anymore then
- Status changed from Feedback to Resolved
Salt special handling removed again...
Also available in: Atom
PDF