Project

General

Profile

Actions

action #139073

closed

ObsRsync plugin needs to support authentication with 2FA size:M

Added by jbaier_cz 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2023-11-03
Due date:
2023-12-01
% Done:

0%

Estimated time:

Description

Motivation

See #138446.

As the current configuration suggests, osd is using the soon to be protected https://api.suse.de/public. The plugin needs to be able to use authentication.

Acceptance criteria

  • AC1: Products are scheduled as in before on o3
  • AC2: No obvious failed minion jobs related to obs_rsync on o3
  • AC3: Same for OSD

Acceptance tests

Out of scope

  • ObsRsync plugin is documented

Suggestions

  • Consider using osc or osc-tiny CLI instead of implementing the protocol
  • Take a look at SCC and osc-tiny implementations for reference
  • Do what's necessary to fix
  • Ensure that osd is able to fetch the necessary data using authenticated query
  • Ensure that o3 is able to fetch the necessary data as before

Related issues 2 (0 open2 closed)

Related to QA - action #138446: Ensure SUSE QE tooling always uses authenticated IBS API access size:MResolvedjbaier_cz2023-10-24

Actions
Related to QA - action #112871: obs_rsync_run Minion tasks fail with no error message size:MResolvedlivdywan

Actions
Actions #1

Updated by jbaier_cz 12 months ago

  • Related to action #138446: Ensure SUSE QE tooling always uses authenticated IBS API access size:M added
Actions #2

Updated by jbaier_cz 12 months ago

  • Target version set to Ready
Actions #3

Updated by livdywan 12 months ago

  • Description updated (diff)
Actions #4

Updated by livdywan 12 months ago

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

Let's remember that this is going to be needed very soon. I assume so far it's not been enforced, or we would have seen problems.

Actions #5

Updated by josegomezr 12 months ago

  • Assignee set to josegomezr

I'm taking this one!

Actions #6

Updated by tinita 12 months ago

Actions #8

Updated by okurz 12 months ago

  • Subject changed from ObsRsync plugin needs to support authentication with 2FA to ObsRsync plugin needs to support authentication with 2FA size:M
  • Description updated (diff)
  • Status changed from New to In Progress
Actions #9

Updated by tinita 12 months ago

https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 - Got credentials from Filip, but not working yet. Waiting until it's fixed

Actions #10

Updated by openqa_review 12 months ago

  • Due date set to 2023-12-01

Setting due date based on mean cycle time of SUSE QE Tools

Actions #11

Updated by livdywan 12 months ago

livdywan wrote in #note-4:

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892

Actions #12

Updated by livdywan 12 months ago

livdywan wrote in #note-11:

livdywan wrote in #note-4:

As part of hardening the security posture of our internal build service, please be notified that we are going to be disabling the anonymous read access to IBS effective November 15th, possibly delayed to November 30 if there's a good reason given.

I don't know where this deadline is being communicated other than our tickets, and I'd rather be explicit here so I added a comment on SD-138892

"It was agreed to postpone the implementation of disabling anonymous access to IBS to November 30th for reasons like this one."

So I take it the deadline was moved up.

Actions #13

Updated by okurz 12 months ago

  • Assignee changed from josegomezr to tinita

As discussed in tools team coordination meeting tinita should take over, waiting for https://sd.suse.com/servicedesk/customer/portal/1/SD-138892 . In the meantime the config can already be prepared, e.g. draft merge request for the openqa.ini for OSD. josegomezr is available to help on request.

EDIT: meanwhile deployed to OSD, also see https://openqa.suse.de/changelog

Actions #14

Updated by tinita 12 months ago

I got new credentials and was able to successfully login to https://idp-mfa.suse.de/ and https://build.suse.de/ .
I created an SSH key and put it into IDP.
Tried with @josegomezr to find out why osc can't authenticate. Investigation still ongoing.

Actions #15

Updated by tinita 12 months ago · Edited

We tried to add several ssh keys to IDP and put them in ~geekotest/.ssh and ~geekotest/.oscrc, but we can't authenticate:

% osc -vdA https://api.suse.de ls
...
Server returned an error: HTTP Error 401: Unauthorized
...
% osc -A https://api.suse.de --http-full-debug --debug --no-keyring api /build/SUSE:Factory:Head
...
Server returned an error: HTTP Error 401: Unauthorized
...
Actions #16

Updated by osukup 12 months ago

tinita wrote in #note-15:

We tried to add several ssh keys to IDP and put them in ~geekotest/.ssh and ~geekotest/.oscrc, but we can't authenticate:

it takes pretty long time to propagate change in IDP :( when I updated my private key I was able authenticate with orc after 24h :(

Actions #17

Updated by tinita 12 months ago

I asked here https://suse.slack.com/archives/C02BX1X92HM/p1700578336235169 but so far noone was able to help.
Should I create a ticket?

Actions #18

Updated by tinita 12 months ago

It's been more than 24h since I put that key in IDP, still not working, so it must be something else.

Actions #19

Updated by tinita 11 months ago

Apparently there was a problem when creating the account, which lead to the 401 Unauthorized.
osc commands work now like expected.
Next step: Enable the new url in the plugin.

Actions #20

Updated by tinita 11 months ago · Edited

Ok, the new url is enabled in /etc/openqa/openqa.ini on osd, and it's working.
Tested with:

MOJO_CLIENT_DEBUG=1 /usr/share/openqa/script/openqa eval -V 'my $x = app->obs_rsync; my $d = $x->is_status_dirty("SUSE:ALP:Source:Standard:1.0:Staging:V", 1); $d'

I had to do one workaround: chown geekotest /var/lib/openqa. It belonged to root.
The current code tries to create a tempfile in this directory. For a fix see:
https://github.com/os-autoinst/openQA/pull/5372 Pass TMPDIR=1 to OBS Rsync authentication

Until then I will keep the directory like that if noone objects.

Actions #21

Updated by okurz 11 months ago

Actions #22

Updated by tinita 11 months ago

  • Status changed from In Progress to Feedback

I cleaned up geekotest's .ssh directory.
/var/lib/openqa belongs to root again, although I didn't do that. Maybe it was done by salt.
AT1-1 and AT3-2 and AT2-1 are looking fine.
For AT3-1 I see a few failures. But I don't know where to look for the actual errors.
The gru journal doesn't show anything related.

Actions #23

Updated by tinita 11 months ago

I looked at one failure:
https://openqa.suse.de/minion/jobs?id=9516755
and found the entry in /var/log/openqa_gru:

[2023-11-28T14:36:13.826482+01:00] [debug] Process 8647 is performing job "9516755" with task "obs_rsync_run"
[2023-11-28T14:36:14.064975+01:00] [error] ObsRsync#_run failed (256): No message
[2023-11-28T14:36:14.069348+01:00] [error] Gru job error: {
  "code" => 256,
  "message" => "No message"
}

which is not really helpful.
Also nothing in here:

# ls -lrt /opt/openqa-trigger-from-ibs/SUSE:ALP:Source:Standard:1.0:Staging:F
total 32
-rw-r--r-- 1 geekotest nogroup    7 Nov 28 14:36 .job_id
-rw-r--r-- 1 geekotest nogroup    9 Nov 28 14:36 .dirty_status
-rw-r--r-- 1 geekotest nogroup    0 Nov 28 14:36 files_iso.lst
-rw-r--r-- 1 geekotest nogroup    7 Nov 28 14:36 .last_failed_job_id
-rw-r--r-- 1 geekotest nogroup 1920 Nov 28 15:33 read_files.sh
-rw-r--r-- 1 geekotest nogroup 2996 Nov 28 15:33 print_rsync_repo.sh
-rw-r--r-- 1 geekotest nogroup 2104 Nov 28 15:33 print_rsync_iso.sh
-rw-r--r-- 1 geekotest nogroup 5053 Nov 28 15:33 print_openqa.sh

So I need help

Actions #24

Updated by okurz 11 months ago

Also message received from ro from BuildOps

hi ... ich sehe hier noch zugriffe von openqa.oqa.prg2.suse.org auf /public im IBS

  • "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:09 +0100] "GET /public/build/SUSE:SLE-15-SP6:GA:Staging:H/_result?package=000product HTTP/1.1" 200 985 "-" "Mojolicious (Perl)" Employee="-"
  • "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:33:21 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
  • "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:34:22 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-"
  • "2a07:de40:b203:12:0:ff:fe4f:7c2b, 192.168.23.1" - [28/Nov/2023:13:35:26 +0100] "GET /public/build/SUSE:SLE-15-SP3:Update:BCI/_result?package=000product HTTP/1.1" 200 2640 "-" "Mojolicious (Perl)" Employee="-" habt ihr da schon code der dann authentifiziert mit dem IBS redet wenn wir /public und anonymous access abschalten ?
Actions #25

Updated by tinita 11 months ago

Oh right, I forgot to change the URL in the o3 config. Did that now...
I restarted the webui.

Actions #26

Updated by tinita 11 months ago

I needed to change the config in salt as well: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1059

Actions #28

Updated by tinita 11 months ago · Edited

We discussed in the unblock that it might be good to extend the default user-agent connection timeout of 10s to 30s, because in the case of authentication requests that can take longer than about 7s (because of ssh key caching).

Actions #29

Updated by tinita 11 months ago

  • Related to action #112871: obs_rsync_run Minion tasks fail with no error message size:M added
Actions #30

Updated by tinita 11 months ago

  • Status changed from Feedback to Resolved

We decided we don't need to increase the connect timeout.

Actions

Also available in: Atom PDF