QA » openQA Project

During the openQA weekly we've talked about this ticket and consider it a good candidate for a mob session. Main problems to solve are Salt deployment and SSL configuration. As well as a simple way to rollback the deployment and use Apache again in case something goes wrong.

We can prepare the deployment of nginx in parallel to apache, have it deployed and at any time decide when to switch by just disabling/enabling services accordingly. The deployment needs to consider dehydrated+nginx as well. We can switch OSD to nginx to gather realtime data before we suggest to use nginx as default in our openQA documentation and CI infrastructure.

It seems to generally work with the config I've already put on Slack: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1168

It uses different ports (the normal ports plus 1000). Therefore I also had to add service_port_delta = 0 to the config to make the live mode work (as it would otherwise assume a reverse-proxy-less development setup).

I'll test connecting a worker after lunch.

I connected a worker via

HOST=https://openqa.suse.de:1443
BACKEND=qemu
WORKER_CLASS=qemu_x86_64_poo130636

and it worked (registration, picking up a job and concluding it).

So I'll prepare a MR to switch ports which we can merge next week. EDIT: https://gitlab.suse.de/mkittler/salt-states-openqa/-/merge_requests/new?merge_request%5Bsource_branch%5D=nginx-for-real

I also went through the web UI just to see if anything stands out. Cloned a bunch of jobs, and it seems fine https://openqa.suse.de:1443/tests/overview?distri=sle&version=15-SP4&build=poo%23130636 - note that the port keeps being reset, even the output of openqa-clone-job --repeat 100 --within-instance https://openqa.suse.de/tests/14196034 _GROUP=0 BUILD=poo#130636 gave me URLs without a port so this may have made manual testing less relevant.

This test is in fact not really relevant as all of those tests probably just ran on a workers that connected via apache. But at least we know that there are no surprises with openqa-clone-job itself (if it actually honored the port).

I created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1169 to use NGINX for real.

I'll have to make a break in two hours so currently there's not a big enough window for me to merge it. So I'll merge it when I get back or tomorrow.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1173 to get the same, faster asset handling as on o3

Even though implementing the monitoring is out of scope we should probably at least get rid of

openqa.suse.de:
    2024-05-08T11:24:18Z E! [inputs.apache] Error in plugin: http://localhost/server-status?auto returned HTTP status 404 Not Found
    2024-05-08T11:24:21Z E! [telegraf] Error running agent: input plugins recorded 1 errors

as it lets our pipelines fail.

EDIT: MRs:

• Disable Apache monitoring: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1177
• Draft for enabling NGINX monitoring: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1178

We also encounter strange failures when clicking on links to YAML schedules in the settings tab. Example:

https://openqa.suse.de/tests/14263283/settings/schedule/yast/maintenance/create_hdd_transactional_server_restapi.yaml

displays the following text instead of the YAML schedule:

File path: /var/lib/openqa/share/tests/sle/schedule/yast/maintenance/create_hdd_transactional_server_restapi.yaml
let mode; let path = document.getElementById('script').dataset.path; if (path && path.endsWith('.pm') || path.endsWith('.pl')) { mode = 'ace/mode/perl'; } var editor = ace.edit("script", { mode: mode, maxLines: Infinity, readOnly: true, }); editor.session.setUseWrapMode(true);

Oh, that seems to be a missing quote of the data-path. Looking at the source:

<div class="code" id="script" data-path="/var/lib/openqa/share/tests/sle/schedule/yast/maintenance/create_hdd_transactional_server_restapi.yaml>---
...
</div>

  <script type="text/javascript">
let mode;
let path = document.getElementById('script').dataset.path;
...
</script>

</div>

Can't see how that's related to the nginx though...

Fix: https://github.com/os-autoinst/openQA/pull/5631

Ok, so nothing problematic came up besides https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1180 which has already been merged.

The two monitoring MRs have been merged as well. Of course Grafana still needs to be adjusted (as the 2nd MR only covers Telegraf) but we defined this out of scope for this ticket.

EDIT: Looks like https://progress.opensuse.org/issues/160171 is related, too.