action #152755
closed[tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? size:M
0%
Description
Observation¶
openQA test in scenario sle-15-SP4-Server-DVD-Updates-x86_64-qam-regression-installation-SLES@64bit fails in
scc_registration
Reproducible¶
Fails since (at least) Build 20231218-1
and seems like all SCC related jobs in at least the same build
https://openqa.suse.de/tests/overview?build=20231218-1&groupid=414&version=15-SP4&distri=sle
are affected
https://openqa.suse.de/tests/13109842#comments
shows 4x red so a clear reproducible infrastructure related issue.
but https://openqa.suse.de/tests/13111926#step/scc_registration/8
Expected result¶
Last good: 20231217-1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Suggestions¶
- SCC is up and running - it's not a general downtime of the service
TID for diagnosing SCC connectivity issues: https://www.suse.com/support/kb/doc/?id=000021034
TL;DR: This bad boi should return a reflection of the HTTP request (minus the post data)curl -sSL -d post=data -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" -H "System-Token: testtoken" https://scc.suse.com/debug/reflect | python3 -m json.tool
Use a developer session to debug the issue live
Updated by livdywan almost 1 year ago
- Subject changed from [tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? to [tools] test fails in scc_registration - SCC not reachable despite not running multi-machine tests? size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by okurz almost 1 year ago
- Status changed from Workable to In Progress
- Assignee set to mkittler
Crosschecking if it's still not the product as I suspect SLE maintenance tests could bring in a dynamic change
openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.suse.de/tests/13085470 {TEST,BUILD}+=-poo152755-okurz _GROUP=0 TESTS=tests/installation/isosize,tests/installation/bootloader,tests/installation/welcome,tests/installation/scc_registration
-> https://openqa.suse.de/tests/13112583
which was picked up by sapworker3 that runs in NUE2.
and specific to worker35 running in PRG2.
openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.suse.de/tests/13085470 {TEST,BUILD}+=-poo152755-okurz _GROUP=0 TESTS=tests/installation/isosize,tests/installation/bootloader,tests/installation/welcome,tests/installation/scc_registration WORKER_CLASS=worker35
Updated by okurz almost 1 year ago
- Related to action #152389: significant increase in MM-test failure ratio 2023-12-11: test fails in multipath_iscsi and other multi-machine scenarios due to MTU size auto_review:"ping with packet size 1350 failed, problems with MTU" size:M added
Updated by okurz almost 1 year ago
- Related to action #151612: [kernel][tools] test fails in suseconnect_scc - SUT times out trying to reach https://scc.suse.com added
Updated by josegomezr almost 1 year ago · Edited
Notes on the investigation:
Tested with SUT on: https://openqa.suse.de/tests/13112301 (
worker36.oqa.prg2.suse.org:5992
)Switched to tty2 (F8 > Control, F8 > Alt, F2 | F8 > Control, F8 > Alt)
curl
against SCC workscurl -sSL -d post=data -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" -H "System-Token: testtoken" https://scc.suse.com/debug/reflect | python3 -m json.tool
- screenshot
- screenshot
Interacting with SCC through SUSEConnect Ruby Layer works:
irb require 'suse/connect' # Reads the credentials file (talks with libsuseconnect) SUSE::Connect::YaST.read_credentials # List activated products (talks with libsuseconnect, libsuseconnect talks to SCC) SUSE::Connect::YaST.status({}).activated_products.map(&:friendly_name)
The error message comes from libzypp on two locations:
We couldn't find an
rpm
database in any of the mount points on/mounts
.
The root rpm database does not have any usable information.The SUT does not look to have any issue with TLS (as demostrated by the
curl
command successfully reaching SCC). Neither the worker (as attempted
by @Marius).
It seems MTU's are not in play in this scenario.
Updated by josegomezr almost 1 year ago
Addenda: Handy links
Code Search on GitHub for YAST:
https://github.com/search?q=org%3Ayast+%22options+for%22+NOT+repo%3Ayast%2Fyast-translations&type=code&p=5
Code Search on GitHub for libzypp & zypper:
add a
.dev
instead of.com
to any GitHub Link and get a vscode-ish experience on your browser, perfect for multiple searches over the same repository.- https://github.dev/SUSE/connect-ng/
- Within the editor you can
Control + Shift + F
and get full repo search.
Updated by okurz almost 1 year ago
- Status changed from In Progress to Blocked
- Priority changed from Urgent to Normal
https://bugzilla.opensuse.org/show_bug.cgi?id=1218203 looks very much related.
Also see
https://suse.slack.com/archives/C02CL8FJ8UF/p1702984730937109
in #discuss-zypp. It is quite likely now that this is a product regression and nothing that we need to follow up with.
@mkittler please add yourself to the bug to follow it and update this progress ticket accordingly when the bug was resolved.
Updated by josegomezr almost 1 year ago
Found a list of packages saved on /mounts/mp_0001/.packages.root
.
and here's a list of libzypp/libcurl packages: https://paste.opensuse.org/pastes/5b7406969179
But there's a probability that those packages are not a reflection of what's available to the installer itself due to the installer self-update
Updated by josegomezr almost 1 year ago · Edited
Another Handy Note:
Do not do this at home, this is pretty much a data exfiltration technique... but if you to extract a small text file/content out of your SUT. You can post it to https://paste.opesuse.org/
like so:
curl -sSLi --form "file=@%filepath%" https://paste.opensuse.org/ | grep location
# or if you're processing the output
cat the.content | curl -sSLi --form "code=<-" https://paste.opensuse.org/ | grep location
if you want a bit more readable output:
curl -sSLi --form "file=@%filepath%" https://paste.opensuse.org/ | grep location | perl -ne '@_ split "/"; $_[-1] =~ s/(...)/\1 /g; print join("/", @_);'
cat the.content | curl -sSLi -F 'code=<-' https://paste.opensuse.org/ | grep location | perl -ne '@_ split "/"; $_[-1] =~ s/(...)/\1 /g; print join("/", @_);'
BEWARE: Be mindful that paste.opensuse.org is a public channel, do not post privileged information.
Updated by livdywan 12 months ago
okurz wrote in #note-7:
https://bugzilla.opensuse.org/show_bug.cgi?id=1218203 looks very much related.
Hello. I can see that since Marcus' update was introduced to the queue and later released, all the runs are green. Is the issue resolved?
Seems there's some movement here. Can be checked at the next opportunity.
Updated by mkittler 11 months ago
- Status changed from Blocked to Resolved
I was on vacation so I haven't followed what was going on. However, now the scenario mentioned in the ticket description looks good again, indeed. The current builds in the same job group look generally good. So I guess this can be considered resolved, indeed.