Project

General

Profile

Actions
Copy link

action #48674

closed

openqaworker3 MM is broken

Added by asmorodskyi about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
nicksinger
Category:
-
Target version:
-
Start date:
2019-03-05
Due date:
% Done:

0%

Estimated time:

Description

systemctl status os-autoinst-openvswitch.service
● os-autoinst-openvswitch.service - os-autoinst openvswitch helper
Loaded: loaded (/etc/systemd/system/os-autoinst-openvswitch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2019-02-25 17:03:34 CET; 1 weeks 0 days ago
Main PID: 1589 (code=exited, status=255)

Feb 25 17:03:32 linux-5h37 systemd[1]: Starting os-autoinst openvswitch helper...
Feb 25 17:03:34 linux-5h37 systemd[1]: Started os-autoinst openvswitch helper.
Feb 25 17:03:34 linux-5h37 os-autoinst-openvswitch[1589]: can't parse bridge local port IP at /usr/lib/os-autoinst/os-autoinst-openvswitch line 37.
Feb 25 17:03:34 linux-5h37 systemd[1]: os-autoinst-openvswitch.service: Main process exited, code=exited, status=255/n/a
Feb 25 17:03:34 linux-5h37 systemd[1]: os-autoinst-openvswitch.service: Unit entered failed state.
Feb 25 17:03:34 linux-5h37 systemd[1]: os-autoinst-openvswitch.service: Failed with result 'exit-code'.

Actions
Copy link
 #1

Updated by coolo about 5 years ago

  • Project changed from openQA Project to openQA Infrastructure
Actions
Copy link
 #3

Updated by nicksinger about 5 years ago

  • Status changed from New to In Progress
  • Assignee set to nicksinger
Actions
Copy link
 #4

Updated by nicksinger about 5 years ago

  • Status changed from In Progress to Feedback

There was some old service-file floating around on this worker:

QA-Power8-5-kvm.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
QA-Power8-4-kvm.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
powerqaworker-qam-1:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
malbec.arch.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
grenache-1.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker8.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker7.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker5.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker2.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker6.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker9.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqa.suse.de:
    Unit os-autoinst-openvswitch.service is not loaded: No such file or directory
openqaworker3.suse.de:
    # /etc/systemd/system/os-autoinst-openvswitch.service
openqaworker-arm-1.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaw1.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaw2.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker-arm-2.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker13.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service

This file had wrong dependencies on network:

[Unit]
Description=os-autoinst openvswitch helper
BindsTo=openvswitch.service
After=openvswitch.service
Before=openqa-worker.target salt-minion.service SuSEfirewall2.service

I had the impression of wrong deps after restarting the service by hand. Before it showed the auto-generated hostname in the logs:

 Feb 25 17:03:32 linux-5h37 systemd[1]: Starting os-autoinst openvswitch helper...

While it shows the actual hostname after a manual restart:

Mar 06 10:00:37 openqaworker3 systemd[1]: Starting os-autoinst openvswitch helper...

and indeed, a systemctl cat os-autoinst-openvswitch showed that it actually uses /etc/systemd/system/os-autoinst-openvswitch.service instead of /usr/lib/systemd/system/os-autoinst-openvswitch.service.
I've moved the old service-file away to /root/poo48674_old_service.os-autoinst-openvswitch.service and did a systemctl daemon-reload and restarted the service once again to have the new service startup this time.
Now every worker is using exactly the same service file:

openqa:~ # salt '*' cmd.run 'systemctl cat os-autoinst-openvswitch.service | head -n 1'
QA-Power8-5-kvm.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
malbec.arch.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
powerqaworker-qam-1:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
QA-Power8-4-kvm.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
grenache-1.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker8.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker6.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker2.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker7.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker9.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker5.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqa.suse.de:
    Unit os-autoinst-openvswitch.service is not loaded: No such file or directory
openqaworker3.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker-arm-1.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaw2.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaw1.qa.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker-arm-2.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
openqaworker13.suse.de:
    # /usr/lib/systemd/system/os-autoinst-openvswitch.service
Actions
Copy link
 #5

Updated by nicksinger about 5 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link

Also available in: Atom PDF