action #139073
closedObsRsync plugin needs to support authentication with 2FA size:M
Description
Motivation¶
See #138446.
As the current configuration suggests, osd is using the soon to be protected https://api.suse.de/public
. The plugin needs to be able to use authentication.
Acceptance criteria¶
- AC1: Products are scheduled as in before on o3
- AC2: No obvious failed minion jobs related to obs_rsync on o3
- AC3: Same for OSD
Acceptance tests¶
- AT1-1: Check https://openqa.opensuse.org/admin/productlog for openQA builds triggered correctly, e.g. daily Tumbleweed and Leap snapshots showing up at all
- AT2-1: Check https://openqa.opensuse.org/minion/jobs?state=failed&task=obs_rsync_run for obvious related failed minion jobs
- AT3-1: Check https://openqa.suse.de/minion/jobs?state=failed&task=obs_rsync_run for obvious related failed minion jobs
- AT3-2: Check https://openqa.suse.de/admin/productlog for openQA builds triggered correctly, e.g. latest SLE and ALP snapshots
Out of scope¶
- ObsRsync plugin is documented
Suggestions¶
Updated by jbaier_cz over 1 year ago
- Related to action #138446: Ensure SUSE QE tooling always uses authenticated IBS API access size:M added
Updated by livdywan over 1 year ago
As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.
Let's remember that this is going to be needed very soon. I assume so far it's not been enforced, or we would have seen problems.
Updated by tinita over 1 year ago
I created https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 for a new bot account
Updated by josegomezr over 1 year ago
Here's the PR: https://github.com/os-autoinst/openQA/pull/5360
Updated by okurz over 1 year ago
- Subject changed from ObsRsync plugin needs to support authentication with 2FA to ObsRsync plugin needs to support authentication with 2FA size:M
- Description updated (diff)
- Status changed from New to In Progress
Updated by tinita over 1 year ago
https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 - Got credentials from Filip, but not working yet. Waiting until it's fixed
Updated by openqa_review over 1 year ago
- Due date set to 2023-12-01
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan over 1 year ago
livdywan wrote in #note-4:
As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.
I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892
Updated by livdywan over 1 year ago
livdywan wrote in #note-11:
livdywan wrote in #note-4:
As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.
I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892
"It was agreed to postpone the implementation of disabling anonymous access to IBS to November 30th for reasons like this one."
So I take it the deadline was moved up.
Updated by okurz over 1 year ago
- Assignee changed from josegomezr to tinita
As discussed in tools team coordination meeting tinita should take over, waiting for https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 . In the meantime the config can already be prepared, e.g. draft merge request for the openqa.ini for OSD. josegomezr is available to help on request.
EDIT: meanwhile deployed to OSD, also see https://openqa.suse.de/changelog
Updated by tinita over 1 year ago
I got new credentials and was able to successfully login to https://idp-mfa.suse.de/ and https://build.suse.de/ .
I created an SSH key and put it into IDP.
Tried with @josegomezr to find out why osc can't authenticate. Investigation still ongoing.
Updated by tinita over 1 year ago · Edited
We tried to add several ssh keys to IDP and put them in ~geekotest/.ssh
and ~geekotest/.oscrc
, but we can't authenticate:
% osc -vdA https://api.suse.de ls
...
Server returned an error: HTTP Error 401: Unauthorized
...
% osc -A https://api.suse.de --http-full-debug --debug --no-keyring api /build/SUSE:Factory:Head
...
Server returned an error: HTTP Error 401: Unauthorized
...
Updated by osukup over 1 year ago
tinita wrote in #note-15:
We tried to add several ssh keys to IDP and put them in
~geekotest/.ssh
and~geekotest/.oscrc
, but we can't authenticate:
it takes pretty long time to propagate change in IDP :( when I updated my private key I was able authenticate with orc after 24h :(
Updated by tinita over 1 year ago
I asked here https://suse.slack.com/archives/C02BX1X92HM/p1700578336235169 but so far noone was able to help.
Should I create a ticket?
Updated by tinita over 1 year ago
It's been more than 24h since I put that key in IDP, still not working, so it must be something else.
Updated by tinita over 1 year ago
Apparently there was a problem when creating the account, which lead to the 401 Unauthorized.
osc commands work now like expected.
Next step: Enable the new url in the plugin.
Updated by tinita over 1 year ago · Edited
Ok, the new url is enabled in /etc/openqa/openqa.ini
on osd, and it's working.
Tested with:
MOJO_CLIENT_DEBUG=1 /usr/share/openqa/script/openqa eval -V 'my $x = app->obs_rsync; my $d = $x->is_status_dirty("SUSE:ALP:Source:Standard:1.0:Staging:V", 1); $d'
I had to do one workaround: chown geekotest /var/lib/openqa
. It belonged to root.
The current code tries to create a tempfile in this directory. For a fix see:
https://github.com/os-autoinst/openQA/pull/5372 Pass TMPDIR=1 to OBS Rsync authentication
Until then I will keep the directory like that if noone objects.
Updated by okurz over 1 year ago
https://github.com/os-autoinst/openQA/pull/5372 merged, what's next?
Updated by tinita over 1 year ago
- Status changed from In Progress to Feedback
I cleaned up geekotest's .ssh
directory.
/var/lib/openqa
belongs to root again, although I didn't do that. Maybe it was done by salt.
AT1-1 and AT3-2 and AT2-1 are looking fine.
For AT3-1 I see a few failures. But I don't know where to look for the actual errors.
The gru journal doesn't show anything related.
Updated by tinita over 1 year ago
I looked at one failure:
https://openqa.suse.de/minion/jobs?id=9516755
and found the entry in /var/log/openqa_gru
:
[2023-11-28T14:36:13.826482+01:00] [debug] Process 8647 is performing job "9516755" with task "obs_rsync_run"
[2023-11-28T14:36:14.064975+01:00] [error] ObsRsync#_run failed (256): No message
[2023-11-28T14:36:14.069348+01:00] [error] Gru job error: {
"code" => 256,
"message" => "No message"
}
which is not really helpful.
Also nothing in here:
# ls -lrt /opt/openqa-trigger-from-ibs/SUSE:ALP:Source:Standard:1.0:Staging:F
total 32
-rw-r--r-- 1 geekotest nogroup 7 Nov 28 14:36 .job_id
-rw-r--r-- 1 geekotest nogroup 9 Nov 28 14:36 .dirty_status
-rw-r--r-- 1 geekotest nogroup 0 Nov 28 14:36 files_iso.lst
-rw-r--r-- 1 geekotest nogroup 7 Nov 28 14:36 .last_failed_job_id
-rw-r--r-- 1 geekotest nogroup 1920 Nov 28 15:33 read_files.sh
-rw-r--r-- 1 geekotest nogroup 2996 Nov 28 15:33 print_rsync_repo.sh
-rw-r--r-- 1 geekotest nogroup 2104 Nov 28 15:33 print_rsync_iso.sh
-rw-r--r-- 1 geekotest nogroup 5053 Nov 28 15:33 print_openqa.sh
So I need help
Updated by okurz over 1 year ago
Also message received from ro from BuildOps
hi ... ich sehe hier noch zugriffe von openqa.oqa.prg2.suse.org auf /public im IBS
- "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:09 +0100] "GET /public/build/SUSE:SLE-15-SP6:GA:Staging:H/_result?package=000product HTTP/1.1" 200 985 "-" "Mojolicious (Perl)" Employee="-"
- "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:21 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
- "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:34:22 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
- "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:35:26 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-" habt ihr da schon code der dann authentifiziert mit dem IBS redet wenn wir /public und anonymous access abschalten ?
Updated by tinita over 1 year ago
Oh right, I forgot to change the URL in the o3 config. Did that now...
I restarted the webui.
Updated by tinita over 1 year ago
I needed to change the config in salt as well: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1059
Updated by tinita over 1 year ago
Updated by tinita over 1 year ago · Edited
We discussed in the unblock that it might be good to extend the default user-agent connection timeout of 10s to 30s, because in the case of authentication requests that can take longer than about 7s (because of ssh key caching).
Updated by tinita over 1 year ago
- Related to action #112871: obs_rsync_run Minion tasks fail with no error message size:M added
Updated by tinita over 1 year ago
- Status changed from Feedback to Resolved
We decided we don't need to increase the connect timeout.