QA (public) » openQA Project (public) » openQA Infrastructure (public)

As can be seen https://monitor.qa.suse.de/d/GDbaremetal-support-prg2/dashboard-for-baremetal-support-prg2 we now have a dashboard without data because the hostname and minion nodename are different.

See also https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3443316#L430 which might be related as we discussed:

worker39.oqa.prg2.suse.org:
    Data failed to compile:
----------
    Detected conflicting IDs, SLS IDs need to be globally unique.
    The conflicting ID is 'net.ipv6.conf.br0.accept_ra' and is found in SLS 'base:network.accept_ra' and SLS 'base:openqa.worker'
mania.qe.nue2.suse.org:
    Data failed to compile:
----------
    Detected conflicting IDs, SLS IDs need to be globally unique.
    The conflicting ID is 'net.ipv6.conf.br0.accept_ra' and is found in SLS 'base:network.accept_ra' and SLS 'base:openqa.worker'
ada.qe.prg2.suse.org:

Ensure that monitoring data on https://monitor.qa.suse.de/ continues (not that new dashboards are generated for every host)

this is directly contradicting with the ticket itself. We use the nodename as dashboard identifier (UID getting filled by a for-loop using the nodename) and also instruct telegraf to collect data with only the hostname/nodename (https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/monitoring/telegraf/telegraf-worker.conf#L14-15 -> […] if empty use os.Hostname(). from telegraf docs)

So the ticket currently suggests that we redefine "nodename" (or better: the hostname) as non-unique in our infrastructure. IMHO this task is rather an epic and should be planned/handled accordingly. Also remember that we do this purely because of https://progress.opensuse.org/issues/168811#note-36 - we can also just rename this single machine for now.

nicksinger wrote in #note-10:

[...] Also remember that we do this purely because of #168811-36 - we can also just rename this single machine for now.

well, that's what I am recurringly suggesting. That fulfills all criteria

I changed the hostname manually

iob@baremetal-support:~> sudo hostnamectl set-hostname baremetal-support-prg2
iob@baremetal-support:~> cat /etc/salt/grains
cat: /etc/salt/grains: Permission denied
iob@baremetal-support:~> sudo cat /etc/salt/grains
nodename: baremetal-support-prg2
iob@baremetal-support:~> sudo cat /etc/hostname 
baremetal-support-prg2

Then I updated the openqa/workerconf.sls
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/960

I see reference in salt-states-openqa/monitoring/grafana/alerts_to_delete.yaml but I am not sure if I have to do something there(I will look at it possible on Monday)
Finally I dont see any entry in salt repo other than ./salt/profile/dns/files/prg2_suse_org/dns-qa.suse.de.zone:34:baremetal-support CNAME baremetal-support.qe.nue2.suse.org.. I guess I will have to update this line too? or do I have to add it to the host.yaml in that repo?

I think the alias has to change in the salt/profile/dns/files/prg2_suse_org/dns-qa.suse.de.zone as well

https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/5952

Can someone let me know if this is correct and the only change required in the dns files?

Please introduce a new A-record and add back a CNAME, e.g. in salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org

- baremetal-support            A     10.144.110.162
+ baremetal-support-prg2       A     10.144.110.162
+ baremetal-support            CNAME baremetal-support-prg2.qe.prg2.suse.org.
- salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support            AAAA  2a07:de40:b211:24::162
+ salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support-prg2       AAAA  2a07:de40:b211:24::162

okurz wrote in #note-15:

Please introduce a new A-record and add back a CNAME, e.g. in salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org

- baremetal-support            A     10.144.110.162
+ baremetal-support-prg2       A     10.144.110.162
+ baremetal-support            CNAME baremetal-support-prg2.qe.prg2.suse.org.
- salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support            AAAA  2a07:de40:b211:24::162
+ salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support-prg2       AAAA  2a07:de40:b211:24::162

PR updated

okurz wrote in #note-15:

Please introduce a new A-record and add back a CNAME, e.g. in salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org

- baremetal-support            A     10.144.110.162
+ baremetal-support-prg2       A     10.144.110.162
+ baremetal-support            CNAME baremetal-support-prg2.qe.prg2.suse.org.
- salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support            AAAA  2a07:de40:b211:24::162
+ salt/profile/dns/files/prg2_suse_org/dns-qe.prg2.suse.org:baremetal-support-prg2       AAAA  2a07:de40:b211:24::162

ybonatakis wrote in #note-22:

I found that the hostname was baremetal-support again.
So I updated again with hostnamectl. but hostname --fqdn was failing to give me the long domain name.
As such, I editted the /etc/hosts (and restarted systemd-hostnamed). With the changes in /etc/hosts the fqdn looked fine.
But still dont see any updates in the dashboard.

Please revert your changes to /etc/hosts. hostnamectl shows the problem. Only the transient hostname is set to baremetal-support-prg2, the static hostname is still baremetal-support. I suggest to rename the VM in OpenPlatform itself to be baremetal-support-prg2. Maybe the static hostname is set from the VM name. Also please ensure this is safe over reboots.

Checking the logs (journalctl -fu grafana-server) but I dont know if they are relevant.

Not relevant

So first I assume that the /etc/hosts should be updated somewhere in one of the salt repos. What else should I check out to make sure that the dashboard is properly configured?

Please read
https://gitlab.suse.de/openqa/salt-states-openqa/#setup-production-machine

/etc/salt/minion_id should be the hostname, right now it's the old FQDN.

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/960 merged