QA (public) » openQA Project (public)

Setting due date based on mean cycle time of SUSE QE Tools

As we would have to use a 15.6 with both firewalld and kernel-default from 15.5,
I don't see much value in moving to 15.6 for now.

Let's block this ticket on the bugzilla issue.

It seems nobody else could reproduce the problem yet. dheidler found a way to reproduce and will update https://bugzilla.suse.com/show_bug.cgi?id=1227616 with better steps to reproduce

Setting due date based on mean cycle time of SUSE QE Tools

It seems either you or ybonatakis also removed the package lock on all other workers as well causing issues on bootup of the machines with os-autoinst-openvswitch-service failing and triggering alerts as observed on https://monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1

From w29 journalctl -u os-autoinst-opensvswitch-service

-- Boot 1de0244017dc4cc3aa72fb4c6e93596a --
Dec 29 03:35:15 worker29 systemd[1]: Starting os-autoinst openvswitch helper...
Dec 29 03:35:15 worker29 systemd[1]: Started os-autoinst openvswitch helper.
Dec 29 03:35:16 worker29 os-autoinst-openvswitch[3741]: Waiting for IP on bridge 'br1', 1200s left ...
…
Dec 29 03:55:19 worker29 os-autoinst-openvswitch[3741]: Waiting for IP on bridge 'br1', 3s left ...
Dec 29 03:55:20 worker29 os-autoinst-openvswitch[3741]: Waiting for IP on bridge 'br1', 2s left ...
Dec 29 03:55:21 worker29 os-autoinst-openvswitch[3741]: can't parse bridge local port IP at /usr/lib/os-autoinst/script/os-autoinst-openvswitch li>
Dec 29 03:55:21 worker29 os-autoinst-openvswitch[3741]: Waiting for IP on bridge 'br1', 1s left ...
Dec 29 03:55:21 worker29 systemd[1]: os-autoinst-openvswitch.service: Main process exited, code=exited, status=255/EXCEPTION
Dec 29 03:55:21 worker29 systemd[1]: os-autoinst-openvswitch.service: Failed with result 'exit-code'.
Dec 29 03:55:21 worker29 systemd[1]: os-autoinst-openvswitch.service: Consumed 6.047s CPU time.
Dec 29 04:57:38 worker29 systemd[1]: Starting os-autoinst openvswitch helper...
Dec 29 04:57:38 worker29 systemd[1]: Started os-autoinst openvswitch helper.

so apparently after 1.5h(!) the service ends up fine, probably being auto-restarted after other services in the network stack. Added according rollback step to ticket.

Replace firewalld on osd workers with 20 lines of bash / systemd unit code:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1335

As worker39 failed OSD deployment the second day in a row (https://gitlab.suse.de/openqa/osd-deployment/-/jobs/3631485) I now removed it from salt following https://progress.opensuse.org/projects/openqav3/wiki#Take-machines-out-of-salt-controlled-production - please make sure to revert before closing this ticket

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1335 merged now. I applied several smaller fix-ups afterwards:

• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1344
• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1345
• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1346
• https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1348

CI Deployment maybe did not run on all hosts yet until https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1349 is merged but salt '*' state.apply… on non-wireguard-hosts should work.

Arguably it's infra (and as before we let the assignee choose if it is unclear).

I now pulled out the changes regarding OSD into a separate ticket #175956

ariel:~ # salt-ssh '*' cmd.run 'systemctl is-active nftables && systemctl is-enabled nftables'
openqaworker21:
    active
    enabled
openqaworker28:
    active
    enabled
openqaworker25:
    active
    enabled
openqaworker27:
    active
    enabled
openqaworker20:
    active
    enabled
openqaworker22:
    active
    enabled
openqaworker24:
    active
    enabled
openqaworker26:
    active
    enabled
openqaworker23:
    active
    enabled
openqaworker-arm21:
    active
    enabled
openqaworker-arm22:
    active
    enabled

ariel# salt-ssh '*' cmd.run 'zypper ll'
openqaworker26:

    There are no package locks defined.
openqaworker20:

    There are no package locks defined.
openqaworker21:

    There are no package locks defined.
openqaworker22:

    There are no package locks defined.
openqaworker28:

    There are no package locks defined.
openqaworker24:

    There are no package locks defined.
openqaworker25:

    There are no package locks defined.
openqaworker27:

    There are no package locks defined.
openqaworker23:
    System management is locked by the application with pid 111491 (zypper).
    Close this application before trying again.
openqaworker-arm21:

    There are no package locks defined.
openqaworker-arm22:

    # | Name                   | Type    | Repository | Comment
    --+------------------------+---------+------------+--------
    1 | libply*                | package | (any)      |
    2 | os-autoinst-devel      | package | (any)      |
    3 | xdg-desktop-portal-gtk | package | (any)      |

What's with the two o3 machines in NUE2?

No idea. What are the hostnames?
Also they are not mentioned in AC1.

dheidler wrote in #note-50:

No idea. What are the hostnames?

Please see https://progress.opensuse.org/projects/openqav3/wiki/#Manual-command-execution-on-o3-workers, it's kerosene and aarch64-o3

Also they are not mentioned in AC1.

yes, that was apparently an oversight. But it should be clear that we want a solution for all o3 hosts.

dheidler wrote in #note-32:

Opened https://bugzilla.suse.com/show_bug.cgi?id=1235450

What about this ticket? If we have no ticket blocking on it we won't be keeping track of it. And I don't see it being references in salt 🤔

Please ensure either one is true

This ticket is the one about o3 - so there is no (official) salt.

As the new default is nftables, there is no issue anymore related to our work.
There is only an issue with firewalld, which we don't use anymore.

If you think we should track this, it would be better to create a new ticket about evaluating to switch back to firewalld,
that could be blocked on.

As the ACs of this ticket are fulfilled, there is no reason to block an anything.