action #107857
closedcoordination #104274: [epic] Enable test & release process on the container bot for BCI 15-SP3 images
Enable and fix BCI tests on aarch64 (except dotnet family)
100%
Description
Enable aarch64 hosts for BCI where aarch64 container images will be tested.
Note: dotnet images are only available for x86_64 (donet, aspnet, dotnet-runtime).
Updated by ph03nix over 2 years ago
- Status changed from Workable to In Progress
- Assignee set to ph03nix
Updated by ph03nix about 2 years ago
As a first step I just schedule some aarch64 test runs and we see, how they behave.
I take the 23.45_minial image to do so.
Updated by ph03nix about 2 years ago
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle BUILD=23.45_minimal VERSION=15-SP3 FLAVOR=BCI-Updates CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-minimal:latest BCI_TEST_ENVS=minimal
https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=23.45_minimal&groupid=409
Updated by ph03nix about 2 years ago
The BCI minimal container for aarch64 is exceeding the maximum size in https://openqa.suse.de/tests/8469955#step/minimal/2 https://openqa.suse.de/tests/8469953#step/minimal/2 and https://openqa.suse.de/tests/8469957#step/minimal/2
Updated by ph03nix about 2 years ago
aarch64 images exceeding image size bug: https://bugzilla.suse.com/show_bug.cgi?id=1198081
Updated by ph03nix about 2 years ago
Test run for all architectures: https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=23.45_minimal&groupid=409
Updated by jlausuch about 2 years ago
ph03nix wrote:
Test run for all architectures: https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=23.45_minimal&groupid=409
This is awesome, at least we don't have stupid Host-related or package-related errors.
The tests are running fine :)
Updated by ph03nix about 2 years ago
To check if aarch64 is working, I scheduled now test runs for that platform for all of yesterday's images: https://openqa.suse.de/group_overview/409?limit_builds=50
### Build13.5_dotnet-runtime-6.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=13.5_dotnet-runtime-6.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-runtime:6.0
### Build22.5_dotnet-3.1
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=22.5_dotnet-3.1 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-sdk:3.1
### Build23.28_dotnet-5.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=23.28_dotnet-5.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-sdk:5.0
### Build12.5_dotnet-6.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=12.5_dotnet-6.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-sdk:6.0
### Build24.5_dotnet-runtime-3.1
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=24.5_dotnet-runtime-3.1 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-runtime:3.1
### Build22.28_dotnet-runtime-5.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=22.28_dotnet-runtime-5.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-runtime:5.0
### Build16.5_aspnet-runtime-3.1
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=16.5_aspnet-runtime-3.1 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-aspnet:3.1
### Build15.28_aspnet-runtime-5.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=15.28_aspnet-runtime-5.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-aspnet:5.0
### Build13.5_aspnet-runtime-6.0
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=13.5_aspnet-runtime-6.0 BCI_TEST_ENVS=dotnet CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/dotnet-aspnet:6.0
### Build13.53_nodejs-12-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=13.53_nodejs-12-image BCI_TEST_ENVS=node CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/nodejs:12
###
Build16.54_nodejs-14-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=16.54_nodejs-14-image BCI_TEST_ENVS=node CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/nodejs:14
###
Build12.50_golang-1.16-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=12.50_golang-1.16-image BCI_TEST_ENVS=go CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.16
###
Build11.51_golang-1.17-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=11.51_golang-1.17-image BCI_TEST_ENVS=go CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17
### Build12.53_python-3.6-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=12.53_python-3.6-image BCI_TEST_ENVS=python CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/python:3.6
###
Build13.49_python-3.9-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=13.49_python-3.9-image BCI_TEST_ENVS=python CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/python:3.9
### Build14.43_ruby-2.5-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=14.43_ruby-2.5-image BCI_TEST_ENVS=ruby CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/ruby:2.5
### Build14.99_openjdk-11-devel-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=14.99_openjdk-11-devel-image CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/openjdk-devel:11 BCI_TEST_ENVS=openjdk_devel
### Build14.57_openjdk-11-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=14.57_openjdk-11-image CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/openjdk:11 BCI_TEST_ENVS=openjdk
### Build4.51_nodejs-16-image
openqa-cli api --osd -X POST isos ARCH=aarch64 DISTRI=sle VERSION=15-SP3 FLAVOR=BCI-Updates BUILD=4.51_nodejs-16-image CONTAINER_IMAGE_TO_TEST=registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/nodejs:16 BCI_TEST_ENVS=node
Updated by ph03nix about 2 years ago
I'm seeing some errors and try to understand and reproduce them. Currently working on https://openqa.suse.de/tests/8475450 in la local VM, which is slow due to being a different architecture.
Updated by ph03nix about 2 years ago
I'm seeing errors in the test_popular_npm_repos part of the BCI tests on aarch64. Will now investigate which repos there cause problems.
I need to check which repos cause the issues. First runs on my aarch64 vm on my laptop made more repos fail due to timeouts, but those are false positives. Will use my Raspberry Pi as test system and run the repos in a Leap VM there.
Updated by ph03nix about 2 years ago
Testing in a opensuse/leap container on a aarch64 VM running on a Raspberry Pi¶
Lodash https://github.com/lodash/lodash.git : OK
chalk https://github.com/chalk/chalk.git : OK
Commander.js https://github.com/tj/commander.js.git : OK
Express https://github.com/expressjs/express.git : OK
prop-types https://github.com/facebook/prop-types : FAIL
node-fs-extra https://github.com/jprichardson/node-fs-extra : OK
The prop-types test run fails:
$ npm run lint
> prop-types@15.8.1 lint
> eslint .
/opt/prop-types/prop-types.js
1:622 error Missing space before value for key 'exports' key-spacing
1:833 error Missing space before value for key '1' key-spacing
106:33 error Missing space before value for key './lib/ReactPropTypesSecret' key-spacing
106:47 error Missing space before value for key './lib/has' key-spacing
106:53 error Missing space before value for key '2' key-spacing
173:33 error Missing space before value for key './lib/ReactPropTypesSecret' key-spacing
173:39 error Missing space before value for key '3' key-spacing
785:23 error Missing space before value for key './checkPropTypes' key-spacing
785:54 error Missing space before value for key './lib/ReactPropTypesSecret' key-spacing
785:68 error Missing space before value for key './lib/has' key-spacing
785:86 error Missing space before value for key 'object-assign' key-spacing
785:99 error Missing space before value for key 'react-is' key-spacing
785:106 error Missing space before value for key '4' key-spacing
806:33 error Missing space before value for key './factoryWithThrowingShims' key-spacing
806:63 error Missing space before value for key './factoryWithTypeCheckers' key-spacing
806:76 error Missing space before value for key 'react-is' key-spacing
806:83 error Missing space before value for key '5' key-spacing
820:9 error Missing space before value for key '6' key-spacing
823:9 error Missing space before value for key '7' key-spacing
853:33 error Multiple spaces found before '// eslint-disa...' no-multi-spaces
915:9 error Missing space before value for key '8' key-spacing
1101:9 error Missing space before value for key '9' key-spacing
1286:15 error Missing space before value for key '_process' key-spacing
1286:22 error Missing space before value for key '10' key-spacing
1303:10 error Missing space before value for key '11' key-spacing
1314:36 error Missing space before value for key './cjs/react-is.development.js' key-spacing
1314:73 error Missing space before value for key './cjs/react-is.production.min.js' key-spacing
1314:87 error Missing space before value for key '_process' key-spacing
✖ 28 problems (28 errors, 0 warnings)
28 errors and 0 warnings potentially fixable with the `--fix` option.
The issue is however also present on x86_64
Updated by ph03nix about 2 years ago
The error might be even easier than we thought:
Error: Timeout of 2000ms exceeded
Updated by ph03nix about 2 years ago
6 test runs for NodeJS all failing with Error: Timeout of 2000ms exceeded in the express test repo - See the 4.51_nodejs-16-image build:
- https://openqa.suse.de/tests/8475450#step/node/6
- https://openqa.suse.de/tests/8475449#step/node/6
- https://openqa.suse.de/tests/8475448#step/node/6
- https://openqa.suse.de/tests/8475447#step/node/6
- https://openqa.suse.de/tests/8475446#step/node/6
- https://openqa.suse.de/tests/8475445#step/node/6
We might need to remove the express repo, at least for aarch64
Updated by ph03nix about 2 years ago
To be sure that this is a timeout due to missing horsepower, I performed now the following tests:
- Use the
registry.suse.com/bci/nodejs:16container on my RPi VM and run the express test: PASSING - Use the
registry.suse.com/bci/nodejs:16container on my emulated aarch64 VM and run the express test: FAIL (timeout)
The test procedure was
git clone https://github.com/expressjs/express.git
cd express
npm config set shrinkwrap false && npm rm --silent --save-dev connect-redis && npm run test-ci && npm run lint
To me it looks very much like a timeout due to a too tight timeout because our aarch64 openQA workers are not fast enough.
Updated by jlausuch about 2 years ago
- Related to action #109638: [bci-tests] Add hability to execute/skip tests depending on the arch added
Updated by jlausuch about 2 years ago
- Subject changed from Enable BCI tests on aarch64 to Enable BCI tests on aarch64 (except dotnet family)
- Description updated (diff)
Updated by jlausuch about 2 years ago
- Related to action #107851: Enable BCI tests on s390x (except dotnet family) added
Updated by jlausuch about 2 years ago
- Related to action #107854: Enable BCI tests on ppc64le (except dotnet family) added
Updated by jlausuch about 2 years ago
We have finally enabled all the architectures. The main issue was found in NodeJS test you are mentioning in this ticket.
Feel free to close this one and track the nodeJS test issue as a specific bug report in separate ticket.
Updated by ph03nix about 2 years ago
Just because two people have been asking independently: BCI aarch64 are not enabled as of now, the previous runs have been scheduled manually.
I'm still working on fixing aarch64 before enabling it in the bot.
Updated by jlausuch about 2 years ago
ph03nix wrote:
Just because two people have been asking independently: BCI aarch64 are not enabled as of now, the previous runs have been scheduled manually.
I'm still working on fixing aarch64 before enabling it in the bot.
It is. And it's my mistake.
I merged https://gitlab.suse.de/qac/container-release-bot/-/merge_requests/40 which adds the feature to exclude the arch from certain images,
so that https://gitlab.suse.de/qac/container-release-bot/-/merge_requests/41 could be merged too using this "feature".
But I thought that aarch64 issue was addressed as a bug.
I will just remove aarch64 from the list again. Sorry about that.
Updated by jlausuch about 2 years ago
It's not disabled again: https://gitlab.suse.de/qac/container-release-bot/-/merge_requests/42
Feel free to enable it again when you feel confident.
My apologies!
Updated by ph03nix about 2 years ago
Waiting for upstream fix: https://github.com/expressjs/express/pull/4887
Updated by ph03nix about 2 years ago
I just run the express.js test in a VM on openqaworker-arm-2 without any issues. The performance there is good enough, although that one step is very close to it's maximum:
✔ should not stack overflow with many registered routes (1209ms)
I'm kinda puzzled, why the openQA test failed with a timeout. Will investigate further.
Updated by ph03nix about 2 years ago
I even run the test within the current registry.suse.com/bci/nodejs:14 podman container without any issues.
Updated by ph03nix about 2 years ago
- Subject changed from Enable BCI tests on aarch64 (except dotnet family) to Enable and fix BCI tests on aarch64 (except dotnet family)
Updated by ph03nix about 2 years ago
Fix in BCI repo: https://gitlab.suse.de/dancermak/bci-tests/-/merge_requests/59
Updated by ph03nix about 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100