QA (public) » openQA Project (public) » openQA Infrastructure (public)

I did the manual steps mentioned again. Will see if the problematic dashboards reappear.

They reappeared at least twice by now.

Our dashboards get generated based on the realtime data present in salt. Sometimes it happens that a host accidentally registers against OSD which can show symptoms like this. However, not this time:

openqa:~ # salt-key -L
Accepted Keys:
QA-Power8-4-kvm.qa.suse.de
QA-Power8-5-kvm.qa.suse.de
grenache-1.qa.suse.de
malbec.arch.suse.de
openqa-monitor.qa.suse.de
openqa.suse.de
openqaworker-arm-1.suse.de
openqaworker-arm-2.suse.de
openqaworker-arm-3.suse.de
openqaworker10.suse.de
openqaworker13.suse.de
openqaworker2.suse.de
openqaworker3.suse.de
openqaworker5.suse.de
openqaworker6.suse.de
openqaworker8.suse.de
openqaworker9.suse.de
Denied Keys:
Unaccepted Keys:
powerqaworker-qam-1
Rejected Keys:

All of these machines are expected. Nothing unusual. Going one step deeper into the mine (baha) where this data is generated: https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/monitoring/grafana.sls#L3
It took me way too long to transform this single line of python into a bash command:

openqa:~ # salt -l error --no-color -C 'openqa.suse.de' mine.get 'roles:worker' 'nodename' 'grain'
openqa.suse.de:
    ----------
    QA-Power8-4-kvm.qa.suse.de:
        QA-Power8-4-kvm
    QA-Power8-5-kvm.qa.suse.de:
        localhost
    grenache-1.qa.suse.de:
        grenache-1
    malbec.arch.suse.de:
        malbec
    openqaworker-arm-1.suse.de:
        openqaworker-arm-1
    openqaworker-arm-2.suse.de:
        openqaworker-arm-2
    openqaworker-arm-3.suse.de:
        openqaworker-arm-3
    openqaworker10.suse.de:
        openqaworker10
    openqaworker13.suse.de:
        openqaworker13
    openqaworker2.suse.de:
        openqaworker2
    openqaworker3.suse.de:
        openqaworker3
    openqaworker5.suse.de:
        openqaworker5
    openqaworker6.suse.de:
        openqaworker6
    openqaworker8.suse.de:
        linux-fwcx
    openqaworker9.suse.de:
        openqaworker9

So openqaworker8.suse.de and QA-Power8-5-kvm.qa.suse.de are the misbehaving hosts. Let's see what I can do about this

I'd say the immediate problem this ticket describes is away for now. However, we might need to follow up with https://progress.opensuse.org/issues/76783 if this persists :(

I hope you agree that it makes sense that we ensure good static hostnames already in salt so I recorded #76786 for this. I still see in https://stats.openqa-monitor.qa.suse.de/alerting/list?state=not_ok the host names "linux-fwcx" and "localhost", maybe you need to call a high state once more? If the unexpected dashboards are gone you can resolve the ticket.

I am trying

sudo salt '*monitor*' state.apply

right now and will check.

This wasn't sufficient. The deployed dashboard template files on the monitor host were fine but the "unknown dashboards" were still there. I manually deleted them in the grafana service instance. This might suffice now :) Setting back to nicksinger as original assignee.

We are back with this problem:

sudo salt -l error --no-color -C 'openqa.suse.de' mine.get 'roles:worker' 'nodename' 'grain'
openqa.suse.de:
    ----------
    QA-Power8-4-kvm.qa.suse.de:
        QA-Power8-4-kvm
    QA-Power8-5-kvm.qa.suse.de:
        QA-Power8-5-kvm
    grenache-1.qa.suse.de:
        grenache-1
    malbec.arch.suse.de:
        malbec
    openqaworker-arm-1.suse.de:
        openqaworker-arm-1
    openqaworker-arm-2.suse.de:
        openqaworker-arm-2
    openqaworker-arm-3.suse.de:
        openqaworker-arm-3
    openqaworker10.suse.de:
        openqaworker10
    openqaworker13.suse.de:
        localhost
    openqaworker2.suse.de:
        linux-1nn1
    openqaworker3.suse.de:
        openqaworker3
    openqaworker5.suse.de:
        openqaworker5
    openqaworker6.suse.de:
        openqaworker6
    openqaworker8.suse.de:
        openqaworker8
    openqaworker9.suse.de:
        linux-q6bp
    powerqaworker-qam-1:
        powerqaworker-qam-1

I assume something must have caused this problem to appear more often lately. Maybe related to #75016 and slow link-up time? What do you think?

raising prio due to #73633#note-37

finished #76786 , crosschecked that all hosts have the correct name. Have removed the wrongly generated dashboard files manually and on osd did

salt --hide-timeout \* saltutil.sync_grains,saltutil.refresh_grains,saltutil.refresh_pillar,mine.update ,,,
salt -l error -C 'G@roles:monitor' state.apply

but that still did find -type f ! -name worker-openqaworker-arm-1.json ! -name worker-malbec.json ! -name worker-grenache-1.json ! -name worker-linux-1nn1.json ! -name worker-openqaworker8.json ! -name worker-openqaworker6.json ! -name worker-QA-Power8-5-kvm.json ! -name worker-openqaworker-arm-3.json ! -name worker-QA-Power8-4-kvm.json ! -name worker-powerqaworker-qam-1.json ! -name worker-localhost.json ! -name worker-openqaworker10.json ! -name worker-linux-q6bp.json ! -name worker-openqaworker-arm-2.json ! -name worker-openqaworker3.json ! -name worker-openqaworker5.json ! -name webui.dashboard.json ! -name webui.services.json ! -name failed_systemd_services.json ! -name automatic_actions.json ! -name job_age.json ! -name openqa_jobs.json ! -name status_overview.json -exec rm {} \;. See the wrong names like "worker-localhost" included.

After a systemctl restart on the affected machines the above worked. I still had to delete the dashboards in the grafana webUI.

That should be enough. As I had already tested that the hostname settings are static I don't expect this issue to reappear – well, not soon at least ;)