QA (public) » openQA Project (public) » openQA Infrastructure (public)

Added https://gitlab.suse.de/qa-sle/baremetal-support-configs/

Currently waiting for https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/5803

Reverse engineered the existing ipxe boot script from http://netboot.qe.prg2.suse.org/kernelqa/ipxe.efi by booting it in qemu

qemu-system-x86_64 -bios /usr/share/qemu/ovmf-x86_64.bin -kernel ipxe.efi

dumping the VM memory and using strings to extract the script.

Blocked until DNS is set up: https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/5803
Also waiting for salt firewall rules via https://sd.suse.com/servicedesk/customer/portal/1/SD-172306

2do afterwards:

• Regenerate ipxe images on kernelqa tftp tree for new baremetal-support url
• Add to salt (Ensure velociraptor still working afterwards)
• Update worker.ini on all prg2 bare metal workers for new url (via https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/937)
• Also still 2do: add to racktables
• IPv6: https://docs.google.com/spreadsheets/d/1M1-uAsNzawPsup-_VFa1qi2NBO1_ItCuxU1kfAe2WGw/edit?gid=775374594#gid=775374594

It took infra only 14 days to merge my MR.
Of course that time didn't prevent them to merge other things so I had to rebase 4 times.

Now we have DNS.

Setting due date based on mean cycle time of SUSE QE Tools

Switched over prg2 workers and openQA instance pointing to baremetal-support.qe.prg2.

Asked Matze for some escalation for the firewall rules.

Multiple delays accumulated also due to #173662 and related impacting us negatively.
Now blocked on https://sd.suse.com/servicedesk/customer/portal/1/SD-175317

https://sd.suse.com/servicedesk/customer/portal/1/SD-175317 was resolved, you verified. What's next?

just ipv6 dns missing.

@dheidler salt minion IDs need to be unique. Now with

baremetal-support.qe.nue2.suse.org
baremetal-support.qe.prg2.suse.org

we have problems like in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1328#note_702464 which need urgent handling. How about renaming the second machine to "baremetal-support2" or "baremetal-support-prg2"?

I guess our minion handling could be improved here.
This should not break things as the minion ids are already unique.

But as a workaround I will rename both prg2 and nue2 baremetal-support minion id to "baremetal-support-nue2" and respective for prg2.
This is not in line with our documentation at https://gitlab.suse.de/openqa/salt-states-openqa though.

done.

ipv6 dns done

okurz wrote:

• AC2: bare-metal openQA tests in PRG2 can run without relying on baremetal-support.qe.nue2.suse.org

please reference at least one verification job showing that working.

https://openqa.suse.de/tests/16174750#step/ipxe_install/1

Did the rename happen? I still see failures in the salt pipeline yesterday: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/3559987

monitor.qe.nue2.suse.org:
    Data failed to compile:
----------
    Rendering SLS 'base:monitoring.grafana' failed: while constructing a mapping
  in "<unicode string>", line 15, column 1:
    grafana:
    ^
found conflicting ID '/var/lib/grafana/dashboards/generic-baremetal-support.json'
  in "<unicode string>", line 921, column 1:
    /var/lib/grafana/dashboards/gene ... 
    ^

baremetal-support.qe.prg2.suse.org:~ # cat /etc/salt/minion_id ; echo
baremetal-support.qe.prg2.suse.org

According to #174610 that's still a problem. As suggested in #168811-22 please rename the hostname of the PRG2 instance and ensure that the deployment works again

baremetal-support-prg2.qe.prg2.suse.org sounds 🤮 and also dns changes would involve infra so it wouldn't be fixed before mid January (if they are fast) as well as rebuilding the ipxe images.

So I added a grain on the machine to set the nodename.
See https://progress.opensuse.org/issues/174610

please stop messing around with workarounds and call the ticket "Resolved". How about you create a ticket to have our salt states work properly with the fqdn instead and record the necessary step to revert your undocumented workaround?