Project

General

Profile

Actions

action #152755

closed

[tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? size:M

Added by okurz about 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Start date:
2023-12-19
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-15-SP4-Server-DVD-Updates-x86_64-qam-regression-installation-SLES@64bit fails in
scc_registration

Reproducible

Fails since (at least) Build 20231218-1
and seems like all SCC related jobs in at least the same build
https://openqa.suse.de/tests/overview?build=20231218-1&groupid=414&version=15-SP4&distri=sle
are affected

https://openqa.suse.de/tests/13109842#comments
shows 4x red so a clear reproducible infrastructure related issue.

but https://openqa.suse.de/tests/13111926#step/scc_registration/8

Expected result

Last good: 20231217-1 (or more recent)

Further details

Always latest result in this scenario: latest

Suggestions

  • SCC is up and running - it's not a general downtime of the service
  • TID for diagnosing SCC connectivity issues: https://www.suse.com/support/kb/doc/?id=000021034
    TL;DR: This bad boi should return a reflection of the HTTP request (minus the post data)

    curl -sSL -d post=data -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" -H "System-Token: testtoken" https://scc.suse.com/debug/reflect | python3 -m json.tool
    
  • Find out if there is a relation to #152389 and #151612

  • Use a developer session to debug the issue live


Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #152389: significant increase in MM-test failure ratio 2023-12-11: test fails in multipath_iscsi and other multi-machine scenarios due to MTU size auto_review:"ping with packet size 1350 failed, problems with MTU" size:MResolvedmkittler2023-12-11

Actions
Related to openQA Tests (public) - action #151612: [kernel][tools] test fails in suseconnect_scc - SUT times out trying to reach https://scc.suse.comResolvedmkittler2023-11-28

Actions
Actions #1

Updated by livdywan about 1 year ago

  • Subject changed from [tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? to [tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #2

Updated by okurz about 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler

Crosschecking if it's still not the product as I suspect SLE maintenance tests could bring in a dynamic change

openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.suse.de/tests/13085470 {TEST,BUILD}+=-poo152755-okurz _GROUP=0 TESTS=tests/installation/isosize,tests/installation/bootloader,tests/installation/welcome,tests/installation/scc_registration

-> https://openqa.suse.de/tests/13112583

which was picked up by sapworker3 that runs in NUE2.

and specific to worker35 running in PRG2.

openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.suse.de/tests/13085470 {TEST,BUILD}+=-poo152755-okurz _GROUP=0 TESTS=tests/installation/isosize,tests/installation/bootloader,tests/installation/welcome,tests/installation/scc_registration WORKER_CLASS=worker35

-> https://openqa.suse.de/tests/13112620

Actions #3

Updated by okurz almost 1 year ago

  • Related to action #152389: significant increase in MM-test failure ratio 2023-12-11: test fails in multipath_iscsi and other multi-machine scenarios due to MTU size auto_review:"ping with packet size 1350 failed, problems with MTU" size:M added
Actions #4

Updated by okurz almost 1 year ago

  • Related to action #151612: [kernel][tools] test fails in suseconnect_scc - SUT times out trying to reach https://scc.suse.com added
Actions #5

Updated by josegomezr almost 1 year ago · Edited

Notes on the investigation:

  • Tested with SUT on: https://openqa.suse.de/tests/13112301 (worker36.oqa.prg2.suse.org:5992)

  • Switched to tty2 (F8 > Control, F8 > Alt, F2 | F8 > Control, F8 > Alt)

  • curl against SCC works

    curl -sSL -d post=data -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" -H "System-Token: testtoken" https://scc.suse.com/debug/reflect | python3 -m json.tool
    
    • screenshot

  • Interacting with SCC through SUSEConnect Ruby Layer works:

    irb
    require 'suse/connect'
    # Reads the credentials file (talks with libsuseconnect)
    SUSE::Connect::YaST.read_credentials
    # List activated products (talks with libsuseconnect, libsuseconnect talks to SCC)
    SUSE::Connect::YaST.status({}).activated_products.map(&:friendly_name)
    
  • The error message comes from libzypp on two locations:

  • We couldn't find an rpm database in any of the mount points on /mounts.
    The root rpm database does not have any usable information.

  • The SUT does not look to have any issue with TLS (as demostrated by the curl
    command successfully reaching SCC). Neither the worker (as attempted
    by @Marius).

It seems MTU's are not in play in this scenario.

Actions #6

Updated by josegomezr almost 1 year ago

Addenda: Handy links

Actions #7

Updated by okurz almost 1 year ago

  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to Normal

https://bugzilla.opensuse.org/show_bug.cgi?id=1218203 looks very much related.

Also see
https://suse.slack.com/archives/C02CL8FJ8UF/p1702984730937109
in #discuss-zypp. It is quite likely now that this is a product regression and nothing that we need to follow up with.

@mkittler please add yourself to the bug to follow it and update this progress ticket accordingly when the bug was resolved.

Actions #8

Updated by josegomezr almost 1 year ago

Found a list of packages saved on /mounts/mp_0001/.packages.root.

and here's a list of libzypp/libcurl packages: https://paste.opensuse.org/pastes/5b7406969179

But there's a probability that those packages are not a reflection of what's available to the installer itself due to the installer self-update

Actions #9

Updated by josegomezr almost 1 year ago · Edited

Another Handy Note:

Do not do this at home, this is pretty much a data exfiltration technique... but if you to extract a small text file/content out of your SUT. You can post it to https://paste.opesuse.org/ like so:

curl -sSLi --form "file=@%filepath%" https://paste.opensuse.org/ | grep location
# or if you're processing the output
cat the.content | curl -sSLi --form "code=<-" https://paste.opensuse.org/ | grep location

if you want a bit more readable output:

curl -sSLi --form "file=@%filepath%" https://paste.opensuse.org/ | grep location | perl -ne '@_ split "/"; $_[-1] =~ s/(...)/\1 /g; print join("/", @_);'

cat the.content | curl -sSLi -F 'code=<-' https://paste.opensuse.org/ | grep location | perl -ne '@_ split "/"; $_[-1] =~ s/(...)/\1 /g; print join("/", @_);'

BEWARE: Be mindful that paste.opensuse.org is a public channel, do not post privileged information.

Actions #10

Updated by livdywan 12 months ago

okurz wrote in #note-7:

https://bugzilla.opensuse.org/show_bug.cgi?id=1218203 looks very much related.

Hello. I can see that since Marcus' update was introduced to the queue and later released, all the runs are green. Is the issue resolved?

Seems there's some movement here. Can be checked at the next opportunity.

Actions #11

Updated by mkittler 11 months ago

  • Status changed from Blocked to Resolved

I was on vacation so I haven't followed what was going on. However, now the scenario mentioned in the ticket description looks good again, indeed. The current builds in the same job group look generally good. So I guess this can be considered resolved, indeed.

Actions

Also available in: Atom PDF