action #162689
closed[sle-micro 6.1] add ppc64le image into testing queue
Added by mloviska 7 months ago. Updated 3 months ago.
0%
Description
We have a bunch of new images that were not built in the previous product versions.
Please add these images into container job group.
SL-Micro.ppc64le-6.1-Default-ppc-512-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Default-ppc-4096-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Default-ppc-512-SelfInstall-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Default-ppc-4096-SelfInstall-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Base-ppc-512-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Base-ppc-4096-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Base-ppc-512-SelfInstall-Build3.1.raw.xz
SL-Micro.ppc64le-6.1-Base-ppc-4096-SelfInstall-Build3.1.raw.xz
Acceptance criteria¶
- AC1: update corresponding job group -> https://gitlab.suse.de/qac/qac-openqa-yaml/.../sle-micro/latest_container_host.yaml
- AC2: set
FIRST_BOOT_CONFIG=combustion+ignitionandHDD_2=ignition.qcow2 - AC3: add
slem_dockerandslem_podmantest suites only
Notes¶
- the Mediums are already present
- images are being synced (IBS sync is done)
- failures in podman|docker test modules will be handled in specific tickets
- images provided are of 2 types, as in the flavors:
512or4096sector disk format.
Files
| clipboard-202409181242-udore.png (25.9 KB) clipboard-202409181242-udore.png | Zypper failure on suseconnect status | mdati, 2024-09-18 10:42 |
Updated by mdati 7 months ago · Edited
Created MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1711 to add tests for each image
Updated by mdati 6 months ago
- Related to action #163451: SLEM 6.1 - ppc64 disk boot fails added
Updated by mdati 6 months ago · Edited
Since ppce64le added to the tests definitions in grp 513 after MR1711, last builds many related tests fail on start up in disk_boot module, with Not a bootable device and other issues. Analysis ongoing.
Updated by mdati 6 months ago
After investigation, the ppc64le images type 4096 result affected by issue, for which there is boot failure and don't reach grub phase.
Bug 1227707 opened:
[Build 3.12] openQA SLEM6.1 ppc64le images fail in booting phase
https://bugzilla.suse.com/show_bug.cgi?id=1227707
Updated by mdati 6 months ago · Edited
At the current investigation/analysis status, tested ppc images on KVM/QEMU ppc64le emulator, we have:
4096 sector size ppc images slem6.1, a.t.m don't result bootable, for which bsc#1227707 open. Needing for specific disk formatting, as by discussion in that ticket, then qemu emulation also would needs for proper additional options to fit that feature: a new poo has been created for that scope: https://progress.opensuse.org/issues/164018
512 sector size ppc images slem 6.1, with ignition/combustion image added too, succeed to boot with grub, but after ignition ok, then fail in combustion phase, apparently for connectivity problems Failed to connect to conncheck.opensuse.org port 80. More analysis and tests proceeding in OSD workers environments.
Finally for 512 images too, a bugzilla ticket opened: bsc#1227967
A discussion threads on topic, possibly helpful, here reported too:
https://suse.slack.com/archives/C02CANHLANP/p1721114662253369?thread_ts=1721046604.182859&cid=C02CANHLANP
Updated by ph03nix 6 months ago
- Has duplicate action #162692: [sle-micro 6.1] add ppc64le image into testing queue added
Updated by slo-gin 4 months ago
This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.
Updated by mdati 4 months ago
- Assignee deleted (
mdati) - Priority changed from High to Normal
This topic ppc64 on SLEM 6.1 was discussed in the morning (usr:ph03nix), the outcome was that currently automated testing is not yet working. There might be a guide provided to us on how we can do manual testing. Therefore we should wait for more documentation.
Updated by mdati 4 months ago · Edited
Current sle-m 6.1 Container host last Build all tests pass, but the ppc64le only don't a.t.m., failing due to 2 main issues occurring in (1) suseconnect_scc and (2) toolbox modules.
So, issues we're working on are:
(1) an unclear state of errors is occurring when SUSEConnect invoked, either in initial status check step or in next transactional-update register step like:
...
transactional-update 4.8.1 started
Options: -n register -r INTERNAL-USE-ONLY-6a5f-b9fc --url http://micro-17.1.proxy.scc.suse.de
Separate /var detected.
2024-09-16 15:54:21 tukit 4.8.1 started
2024-09-16 15:54:21 Options: --discard -c4 open
2024-09-16 15:54:25 Using snapshot 4 as base for new snapshot 5.
2024-09-16 15:54:25 /var/lib/overlay/4/etc
2024-09-16 15:54:25 Syncing /etc of previous snapshot 3 as base into new snapshot "/.snapshots/5/snapshot"
2024-09-16 15:54:25 SELinux is enabled.
ID: 5
2024-09-16 15:54:37 Transaction completed.
2024-09-16 15:54:37 tukit 4.8.1 started
2024-09-16 15:54:37 Options: --discard callext 5 SUSEConnect --root {} -r INTERNAL-USE-ONLY-6a5f-b9fc --url http://micro-17.1.proxy.scc.suse.de
2024-09-16 15:54:52 Executing `SUSEConnect --root /tmp/transactional-update-5pK25m -r INTERNAL-USE-ONLY-6a5f-b9fc --url http://micro-17.1.proxy.scc.suse.de`:
[1mRegistering system to registration proxy http://micro-17.1.proxy.scc.suse.de[0m
Rooted at:/tmp/transactional-update-5pK25m
[1m
Announcing system to http://micro-17.1.proxy.scc.suse.de ...[0m
command '/usr/bin/zypper --root /tmp/transactional-update-5pK25m --disable-repositories --xmlout --non-interactive products -i' failed
Error: zypper returned -1 with '<?xml version='1.0'?>
<stream>' (signal: bus error (core dumped))
2024-09-16 15:54:57 Application returned with exit status 255.
2024-09-16 15:54:58 tukit 4.8.1 started
2024-09-16 15:54:58 Options: --discard close 5
2024-09-16 15:55:08 No changes to the root file system - discarding snapshot.
2024-09-16 15:55:09 Merging changes in /etc into the running system.
2024-09-16 15:55:09 Discarding snapshot 5.
2024-09-16 15:55:13 Transaction completed.
transactional-update finished
3OH0n-0-
Not clear yet the bug, probably due to qemu version. Some tests, stopped at suseconnect_scc module beginning for debugging, when running on VNC some time repeated the SUSEConnect --status-text command, it randomly reported either a valid Not Registered result or some zypper error, same messages as in the above 2 steps, like: 
.
(2) The toolbox image is not fully configured yet, therefore some work neeeded to prepare the HDD file for correct use in the slem 6.1 ppc64le toolbox tests.
Updated by ph03nix 4 months ago
As stated in the https://suse.slack.com/archives/C02CGKBCGT1/p1726658367118569 thread, we think issue 1 (suseconnect) is connected to an outdated qemu version on the openQA workers and should be resolved within the next days by automatic worker updates.
Issue 2: The toolbox container is still in the release pipeline, but for SLEM 6.1 we don't have openQA definitions setup up yet. This is in progress in https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1821
Updated by okurz 4 months ago
ph03nix wrote in #note-20:
As stated in the https://suse.slack.com/archives/C02CGKBCGT1/p1726658367118569 thread, we think issue 1 (suseconnect) is connected to an outdated qemu version on the openQA workers and should be resolved within the next days by automatic worker updates.
Can you elaborate on that? Currently OSD runs mostly on Leap 15.5 which has currently qemu 7.1.0-150500.49.18.1 . Unless there is a pending SLE maintenance update including a new version of qemu I doubt that a new version of qemu will come in soon.
Updated by jlausuch 4 months ago · Edited
okurz wrote in #note-21:
ph03nix wrote in #note-20:
As stated in the https://suse.slack.com/archives/C02CGKBCGT1/p1726658367118569 thread, we think issue 1 (suseconnect) is connected to an outdated qemu version on the openQA workers and should be resolved within the next days by automatic worker updates.
Can you elaborate on that? Currently OSD runs mostly on Leap 15.5 which has currently qemu 7.1.0-150500.49.18.1 . Unless there is a pending SLE maintenance update including a new version of qemu I doubt that a new version of qemu will come in soon.
This https://smelt.suse.de/request/345328/ https://build.suse.de/request/show/345328/ , but it's not tested as incident yet. Not sure how long it will take.
Updated by mdati 3 months ago
- Related to action #167383: test fails in disk_boot for toolbox added
Updated by mdati 3 months ago
- Related to deleted (action #167383: test fails in disk_boot for toolbox)
Updated by mdati 3 months ago
- Blocked by action #167383: test fails in disk_boot for toolbox added
Updated by mdati 3 months ago
- Status changed from Blocked to In Progress
As mentioned in https://progress.opensuse.org/issues/167383#note-12, Resuming work here for Container Hosts group 513, to manage the toolbox tests, I use the ppc64 raw.xz images in the HDD_1 with proper pre-updates:
(a) I added CONTAINER_IMAGE_TO_TEST for sle-m 6.1, to update the toolbox whose default was for v5.5 and
(b) I got the certificates updated by adding HOST_VERSION=6.1, to trigger it in tests/transactional/host_config
But then new issues appeared in https://openqa.suse.de/tests/15641792#step/image_podman/114, therefore to fix the code I created the PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20383, review and more changes going on.
Updated by mdati 3 months ago · Edited
Poo status summary:
Actual status of ppc64le tests in OSD group 513 is: all docker test pass, but podman tests fail waiting bsc bug solution.
Issues:
PR#20383 hosted many changes related to raised issues in podman tests and solution applied, step-by-step, during test cloned in the updated branch for ppc64le in Container Host Sle-micro 6.1. It was closed, opting to split it in more PR-s.
So, running VR was noted, being toolbox the first testing module after image preparation, that ppc64le image in HDD_1 has installed the /usr/bin/toolbox (bash script) with default version sle-m 5.5 instead of 6.1, that has no ppc64le image, then failed. A product bug was opened for this issue bsc1227509 and all podman failing tests tagged with it. Other e sibling tests for other architectures pass using v5.5, having image defined instead.
Then, to proceed on test verifications and fixing, as w.a. used the custom image assignemt: CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/slfo/products/sl-micro/6.1/totest/containerfile/suse/sl-micro/6.1/toolbox:latest
Then new issue raised, missing SUSE SSL certificates on HDD_1 image.
Solution her was to set also HOST_VERSION, to trigger certificates loading from updated routine in host_config, setting HOST_VERSION=6.1.
Then new issue raised, missing any sle-micro dedicated folder in IBS CA url, causing tests still to fail
After some update proposals and discussions, it was clarified the similarity of certificates between all distri-s versions, so opted for a default fallback: PR20424. This allowed toolbox test to pass and proceed VR-s on next modules.
But here another issue raised, the impact of CONTAINER_IMAGE_TO_TEST parameter, set on toolbox image, and its use in tests/containers/image.pm call of get_image_uri, overriding the logic of $image with the unexpected toolbox uri.
A proposal to use a new parameter for toolbox was put forward in PR20435. But then discussed and clarified that in those tests never should occur the custom setting of toolbox, that shall use the proper default, after bug1227509 resolved. So, PR closed .
Finally, to resolve missing default SUSE certificates in ppc64 images, enabling loading certificates, MR1871 created and merged to set vaule 6.1 for all architectures.
Updated by mdati 3 months ago
- Status changed from Feedback to In Progress
Note that the bug 1227509 was closed (invalid), considering not expected any new toolbox image for SLE-M 6.1.
In place, new toobox v5.5 for ppc64le provided Discussion.
Therefore podman/toolbox tests shall rely in that 5.5 image to pass.
For that, merged:
MR https://gitlab.suse.de/qac/container-release-bot/-/merge_requests/318
MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1875
Updated by mdati 3 months ago · Edited
Last update:
with last created toolbox 5.5 ppc64le (note-32), tests pass also for ppc64le in 'SLE Micro Toolbox updates'
https://openqa.suse.de/tests/overview?distri=sle-micro&version=5.5&groupid=451.
After toolbox validation flow completed and released in 'registry.suse.com', also podman tests will pass in 'Container Host'
https://openqa.suse.de/tests/overview?arch=ppc64le&distri=sle-micro&version=6.1&groupid=513