QA

Draft: https://github.com/openSUSE/qem-bot/pull/177

Setting due date based on mean cycle time of SUSE QE Tools

In https://sd.suse.com/servicedesk/customer/portal/1/SD-154403 I was asked for the approval of the buildops team as it is the owner of amqps://rabbit.suse.de. They were not happy with us "abusing shared gitlab resources for this" so I suppose we better not go down that road. I'll setup the daemon on qam2.qe.prg2.suse.org instead. I suppose the only real disadvantage is that the AMQP "job" won't show up alongside the others on GitLab.

I just tried it again to see whether DNS has changed now but it still fails.

I also stopped qem-bot-amqp-watcher.service on qam2.qe.prg2.suse.org again as we're going for openplatform. If that turns out working I'll completely remove the service from qam2.qe.prg2.suse.org.

For now I keep this blocked on #156214.

removing due-date due to block

Deploying this on OpenPlatform was rather simple. There was a little bit of clicking on the web UI involved (to assign resources and download the op-prg2-1-staging.yaml file) and then the following CLI commands did the trick:

cd /hdd/openqa-devel/openplatform
export KUBECONFIG=$PWD/op-prg2-1-staging.yaml
kubectl config view # to check whether the env variable is considered as expected
kubectl get nodes # to check whether the CLI client generally works
kubectl apply -f qem-bot.yaml -n qem-bot # to deploy the workload

For the configuration file qem-bot.yaml, see https://github.com/Martchus/qem-bot/pull/new/openplatform.

Unfortunately it runs into the same (probably firewall-related) issue we saw when trying to run it on GitLab:

$ kubectl logs -f -p qem-bot-7496cb6967-hmvrd -n qem-bot
…
Traceback (most recent call last):
  File "./qem-bot/bot-ng.py", line 7, in <module>
    main()
  File "/qem-bot/openqabot/main.py", line 32, in main
    sys.exit(cfg.func(cfg))
  File "/qem-bot/openqabot/args.py", line 77, in do_amqp
    amqp = AMQP(args)
  File "/qem-bot/openqabot/amqp.py", line 33, in __init__
    self.connection = pika.BlockingConnection(pika.URLParameters(args.url))
  File "/usr/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
    self._impl = self._create_connection(parameters, _impl_class)
  File "/usr/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
    raise self._reap_last_connection_workflow_error(error)
pika.exceptions.AMQPConnectionError

It also didn't help to specify --url amqp://… with the IP (instead of using the domain name and TLS). So maybe we need yet another SD-ticket but I first asked in the existing SD-ticket.

The team is getting annoyed and frustrated by not-the-place, not-the-time arguments and people coming up with weird blockers in https://sd.suse.com/servicedesk/customer/portal/1/SD-154403. I will add an according comment and escalate so that we can use OpenPlatform rather than reverting to manually maintaining virtual machines on other less maintained hypervisors.

I provided a suggestion in https://sd.suse.com/servicedesk/customer/portal/1/SD-154403 and mentioned that with the potential for need of escalation in https://suse.slack.com/archives/C02CANHLANP/p1717755396214829 in case we don't get to clear decisions.

As you guys stated and as the latest comment in the ticket was from Martin Piala we need to act on it to drive the implementation further, i.e. to find a way to allow Openplatform containers to access rabbit.suse.de . Setting this to High as mgriessmeier asked about this.

I think it's a better approach to make the work on https://sd.suse.com/servicedesk/customer/portal/1/SD-154403 more explicit and to ensure we have the necessary requirements fulfilled to be able to run qem-bot. Creating separate dedicated infra ticket.

#163331

Found the conversation https://suse.slack.com/archives/C02BXKBMXNV/p1713446598715539 which Ruediger Oertel and Marius Kittler referred to. Quoting from there:

(Marius Kittler) (3 months ago) Does anybody here know who the owner of https://rabbit.suse.de is? I was asked about it when asking to allow gitlab-runners to access amqps://rabbit.suse.de. It was initially setup by @Dominik Heidler and @coolo but they are not owning it now anymore. Maybe someone on the OBS-side is?
[…] (not quoting the complete discussion for the purpose of privacy
(Marius Kittler) We are trying to connect to amqps://rabbit.suse.de from a GitLab CI job to get openQA-related events. And yes, this is normally not what one would use a CI job for but it'll get the job done. … that is like really wasteful for the gitlab-runner side. a simple python script which then in async does the "reject/approve" stuff after parsing the amqp event is way more efficient
(Hendrik Vogelsang) @Marius Kittler an example would be https://github.com/openSUSE/kurren
(Marcus Rueckert) but abusing gitlab for this is imho not the best way
(Marius Kittler) We already have a service (written in Python) that listens for AMQP events and does the required actions when receiving them. Of course this is abusing GitLab instead of just setting up a server. I don't think that having a long-running GitLab job (I suppose 4 hours is the maximum) is that wasteful, though. (It would not be one job per event/action.) And it would be in-line with our existing GitLab-based setup for performing those and similar actions (on a regular schedule). (edited)
… gitlab-runners are not your replacemnt for setting up a small VM for this […] you should not abuse shared gitlab resources for this. and if you set up a VM for your own gitlab-runner. then you can just run the script there and dont have to funnel it through gitlab
(Georg Pfützenreuter) sounds like a use case for OpenPlatform

However reconsidering right now mkittler and me think that gitlab CI is providing the benefits of reporting, alerting, secrets management over containers in OpenPlatform. Also by now with https://sd.suse.com/servicedesk/customer/portal/1/SD-154403 progressed gitlab CI runners should have access to rabbit.suse.de so we can continue with our original plan.