Project

General

Profile

Actions
Copy link

action #158814

closed

o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm size:M

Added by tinita 19 days ago. Updated 10 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
jbaier_cz
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2024-04-10
Due date:
2024-04-26
% Done:

0%

Estimated time:

Description

Observation

From /var/log/openqa:

[2024-04-10T15:20:04.190828Z] [error] [6XDd4QtKL9Pc] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.

[2024-04-10T15:25:06.279066Z] [error] [UYVJthN6XmUF] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.

[2024-04-10T15:45:04.107327Z] [error] [T4fj_bgqmapR] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.

[2024-04-10T15:50:02.488807Z] [error] [xYSu0-F1l8A6] Active version 24 is greater than the latest version 23 at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Pg.pm line 100.

I can't find that in the osd openqa journal.

Looking at the time, it looks related to #158805

Acceptance criteria

  • AC1: All our services using Minion are restarted automatically as necessary on an update of the perl-Minion package

Suggestions

  • Catch the above error in active openQA components and trigger a self-restart (without causing an endless loop of restarts)
  • Use a systemd path unit to trigger an auto-restart
Actions
Copy link
 #1

Updated by tinita 19 days ago

I just restarted the webui on o3.
It seems it asn't restarted after the perl-Minion update (but still the jquery issue was gone, so I didn't check that it was restarted).

Monitoring the log now..

Actions
Copy link
 #2

Updated by mkittler 18 days ago

  • Subject changed from o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm to o3 logreport: Active version 24 is greater than the latest version 23 at .../Mojo/Pg.pm size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #3

Updated by jbaier_cz 18 days ago

  • Assignee set to jbaier_cz
Actions
Copy link
 #4

Updated by jbaier_cz 18 days ago

If I understood the problem correctly, we want to restart on db migrations, i.e. when /usr/lib/perl5/vendor_perl/5.38.2/Minion/Backend/resources/migrations/pg.sql changes

Actions
Copy link
 #5

Updated by jbaier_cz 18 days ago

Actions
Copy link
 #6

Updated by jbaier_cz 18 days ago

  • Status changed from Workable to In Progress

I installed the changed rpms manually, enable the auto-restart unit via systemctl enable --now openqa-minion-restart.path and after faking Minion update via zypper install --force perl-Minion, I spotted the following:

Apr 11 15:49:33 dagr.qam.suse.cz [RPM][26338]: Transaction ID 6617ea6d started
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Starting Restarts services which are using Minion...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping Handler for live view in openQA's web UI...
Apr 11 15:49:34 dagr.qam.suse.cz openqa-webui-daemon[25830]: [warn] Stopping worker 25984 immediately
...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping The openQA web UI...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopping The openQA daemon for various background tasks like cleanup and saving needles...
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: openqa-livehandler.service: Deactivated successfully.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopped Handler for live view in openQA's web UI.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Started Handler for live view in openQA's web UI.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: openqa-webui.service: Deactivated successfully.
Apr 11 15:49:34 dagr.qam.suse.cz systemd[1]: Stopped The openQA web UI.
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: install perl-Minion-10.290.0-lp155.2.1.noarch: success
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: install perl-Minion-10.290.0-lp155.2.1.noarch: success
Apr 11 15:49:34 dagr.qam.suse.cz [RPM][26338]: Transaction ID 6617ea6d finished: 0
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-gru.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Stopped The openQA daemon for various background tasks like cleanup and saving needles.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Starting Setup local PostgreSQL database for openQA...
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-setup-db.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Finished Setup local PostgreSQL database for openQA.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Started The openQA web UI.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Started The openQA daemon for various background tasks like cleanup and saving needles.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: openqa-minion-restart.service: Deactivated successfully.
Apr 11 15:49:37 dagr.qam.suse.cz systemd[1]: Finished Restarts services which are using Minion.

I believe that's what we want.

Actions
Copy link
 #7

Updated by openqa_review 18 days ago

  • Due date set to 2024-04-26

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #8

Updated by jbaier_cz 14 days ago

The code is merged and deployed. I did systemctl enable --now openqa-minion-restart.path on o3 to enable the service. Do we want that also for osd? If yes: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1155

Actions
Copy link
 #9

Updated by okurz 14 days ago

Don't we want it on all openQA instances? If yes then what change should we need in salt states?

Actions
Copy link
 #10

Updated by livdywan 14 days ago

okurz wrote in #note-9:

Don't we want it on all openQA instances? If yes then what change should we need in salt states?

The salt change just enables and runs the openqa-minion-restart unit

Actions
Copy link
 #11

Updated by livdywan 14 days ago

jbaier_cz wrote in #note-8:

The code is merged and deployed. I did systemctl enable --now openqa-minion-restart.path on o3 to enable the service. Do we want that also for osd? If yes: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1155

I guess this isn't quite working yet... or maybe deployed too early?

      ID: openqa-minion-restart.path
Function: service.running
  Result: False
 Comment: The named service openqa-minion-restart.path is not available
 Started: 00:57:40.673701
Duration: 43.035 ms
 Changes:

https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2500285

Actions
Copy link
 #12

Updated by jbaier_cz 14 days ago

livdywan wrote in #note-11:

jbaier_cz wrote in #note-8:

The code is merged and deployed. I did systemctl enable --now openqa-minion-restart.path on o3 to enable the service. Do we want that also for osd? If yes: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1155

I guess this isn't quite working yet... or maybe deployed too early?

According to 

# ls -l /usr/lib/systemd/system/openqa-minion-restart.path
-rw-r--r-- 1 root root 134 Apr 15 17:46 /usr/lib/systemd/system/openqa-minion-restart.path

that file should be deployed, so it might be a salt thing not understanding I do not want a .service file but a .path unit file

Actions
Copy link
 #13

Updated by livdywan 14 days ago ยท Edited 

# ls -l /usr/lib/systemd/system/openqa-minion-restart.path
-rw-r--r-- 1 root root 134 Apr 15 17:46 /usr/lib/systemd/system/openqa-minion-restart.path

that file should be deployed, so it might be a salt thing not understanding I do not want a .service file but a .path unit file

Checking the docs, neither salt.states.service nor salt.modules.systemd_service says anything about path units. The implementation seems to recognize it, though?

Actions
Copy link
 #14

Updated by jbaier_cz 11 days ago

  • Status changed from In Progress to Resolved

Whatever was the reason, it seems it was a temporary issue, second run of the deploy job succeeded. With this, I guess we are done here?

Actions
Copy link
 #15

Updated by okurz 11 days ago

  • Status changed from Resolved to Feedback

As discussed in the coord call please pull in the path unit as dependency automatically already in upstream and then remove the special handling in salt again as IIUC that would not be needed anymore then

Actions
Copy link
 #17

Updated by jbaier_cz 10 days ago

  • Status changed from Feedback to Resolved

Salt special handling removed again...

Actions
Copy link

Also available in: Atom PDF