QA » openQA Project

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

Let's remember that this is going to be needed very soon. I assume so far it's not been enforced, or we would have seen problems.

I created https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 for a new bot account

Here's the PR: https://github.com/os-autoinst/openQA/pull/5360

https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 - Got credentials from Filip, but not working yet. Waiting until it's fixed

livdywan wrote in #note-4:

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892

livdywan wrote in #note-11:

livdywan wrote in #note-4:

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892

"It was agreed to postpone the implementation of disabling anonymous access to IBS to November 30th for reasons like this one."

So I take it the deadline was moved up.

I got new credentials and was able to successfully login to https://idp-mfa.suse.de/ and https://build.suse.de/ .
I created an SSH key and put it into IDP.
Tried with @josegomezr to find out why osc can't authenticate. Investigation still ongoing.

We tried to add several ssh keys to IDP and put them in ~geekotest/.ssh and ~geekotest/.oscrc, but we can't authenticate:

% osc -vdA https://api.suse.de ls
...
Server returned an error: HTTP Error 401: Unauthorized
...
% osc -A https://api.suse.de --http-full-debug --debug --no-keyring api /build/SUSE:Factory:Head
...
Server returned an error: HTTP Error 401: Unauthorized
...

tinita wrote in #note-15:

We tried to add several ssh keys to IDP and put them in ~geekotest/.ssh and ~geekotest/.oscrc, but we can't authenticate:

it takes pretty long time to propagate change in IDP :( when I updated my private key I was able authenticate with orc after 24h :(

I asked here https://suse.slack.com/archives/C02BX1X92HM/p1700578336235169 but so far noone was able to help.
Should I create a ticket?

It's been more than 24h since I put that key in IDP, still not working, so it must be something else.

Apparently there was a problem when creating the account, which lead to the 401 Unauthorized.
osc commands work now like expected.
Next step: Enable the new url in the plugin.

Ok, the new url is enabled in /etc/openqa/openqa.ini on osd, and it's working.
Tested with:

MOJO_CLIENT_DEBUG=1 /usr/share/openqa/script/openqa eval -V 'my $x = app->obs_rsync; my $d = $x->is_status_dirty("SUSE:ALP:Source:Standard:1.0:Staging:V", 1); $d'

I had to do one workaround: chown geekotest /var/lib/openqa. It belonged to root.
The current code tries to create a tempfile in this directory. For a fix see:
https://github.com/os-autoinst/openQA/pull/5372 Pass TMPDIR=1 to OBS Rsync authentication

Until then I will keep the directory like that if noone objects.

https://github.com/os-autoinst/openQA/pull/5372 merged, what's next?

I looked at one failure:
https://openqa.suse.de/minion/jobs?id=9516755
and found the entry in /var/log/openqa_gru:

[2023-11-28T14:36:13.826482+01:00] [debug] Process 8647 is performing job "9516755" with task "obs_rsync_run"
[2023-11-28T14:36:14.064975+01:00] [error] ObsRsync#_run failed (256): No message
[2023-11-28T14:36:14.069348+01:00] [error] Gru job error: {
  "code" => 256,
  "message" => "No message"
}

which is not really helpful.
Also nothing in here:

# ls -lrt /opt/openqa-trigger-from-ibs/SUSE:ALP:Source:Standard:1.0:Staging:F
total 32
-rw-r--r-- 1 geekotest nogroup    7 Nov 28 14:36 .job_id
-rw-r--r-- 1 geekotest nogroup    9 Nov 28 14:36 .dirty_status
-rw-r--r-- 1 geekotest nogroup    0 Nov 28 14:36 files_iso.lst
-rw-r--r-- 1 geekotest nogroup    7 Nov 28 14:36 .last_failed_job_id
-rw-r--r-- 1 geekotest nogroup 1920 Nov 28 15:33 read_files.sh
-rw-r--r-- 1 geekotest nogroup 2996 Nov 28 15:33 print_rsync_repo.sh
-rw-r--r-- 1 geekotest nogroup 2104 Nov 28 15:33 print_rsync_iso.sh
-rw-r--r-- 1 geekotest nogroup 5053 Nov 28 15:33 print_openqa.sh

So I need help

Also message received from ro from BuildOps

hi ... ich sehe hier noch zugriffe von openqa.oqa.prg2.suse.org auf /public im IBS

• "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:09 +0100] "GET /public/build/SUSE:SLE-15-SP6:GA:Staging:H/_result?package=000product HTTP/1.1" 200 985 "-" "Mojolicious (Perl)" Employee="-"
• "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:21 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
• "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:34:22 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
• "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:35:26 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-" habt ihr da schon code der dann authentifiziert mit dem IBS redet wenn wir /public und anonymous access abschalten ?

Oh right, I forgot to change the URL in the o3 config. Did that now...
I restarted the webui.

I needed to change the config in salt as well: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1059

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1059 merged

We discussed in the unblock that it might be good to extend the default user-agent connection timeout of 10s to 30s, because in the case of authentication requests that can take longer than about 7s (because of ssh key caching).