action #60626
closed"Download .* processed" but ending up with setup-failure, workers are missing updated perl-Mojolicious >= 8.24
0%
Description
Since today's deployment on osd we have many incompletes like https://openqa.suse.de/tests/3661916
[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] +++ setup notes +++
[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] Start time: 2019-12-04 10:24:09
[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] Running on malbec:2 (Linux 4.12.14-lp151.27-default #1 SMP Fri May 10 14:13:15 UTC 2019 (862c838) ppc64le)
[2019-12-04T11:24:09.0256 CET] [debug] [pid:25781] Downloading sle-12-SP2-ppc64le-4.4.121-190.1.gd3cd4b0-Server-DVD-Incidents-Kernel@ppc64le-virtio-with-ltp.qcow2, request #43477 sent to Cache Service
[2019-12-04T11:24:14.0320 CET] [debug] [pid:25781] Download of sle-12-SP2-ppc64le-4.4.121-190.1.gd3cd4b0-Server-DVD-Incidents-Kernel@ppc64le-virtio-with-ltp.qcow2 processed
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] +++ worker notes +++
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] End time: 2019-12-04 10:24:14
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] Result: setup failure
[2019-12-04T11:24:14.0355 CET] [info] [pid:48327] Uploading autoinst-log.txt
triggered most likely by one of the following changes in openQA-worker:
- Increase SQLite busy timeout and reduce the number of concurrent SQLite connections
- Show log content in test details tab on no module results
- Fix asset definition of SVG icon for running jobs
- Check generate-documentation if target branch exists
- Calculate coverage explicitly in Selenium tests
- Improve group property HTML template
- travis_test: Add requirements for spellcheck in os-autoinst
- Use the cache tmp directory for temporary download files created by the cache service
- Use info instead of warn messages (so monitoring does not get triggered) and quote all string values
- Allow for very quiet cache service tests
- Be a little less verbose with init log messages
- Allow from_worker to set attributes as well
- Remove redundant error message
- Format cache service logs to start with a newline
- Improve cache service log messages to help with debugging issues of missing assets
- Log all cache service output on both sides (cache service and worker) and fix many small test issues
- Handle cache model logging through an attribute
Problem¶
sriedel: Got it, the Mojolicious version is somehow not high enough on openqaworker6
okurz: all osd workers
sriedel: it's probably higher on O3, that's why everything worked fine there we need perl-Mojolicious 8.24
okurz: we have the same on all osd and it's 8.18, on o3 workers it's 8.26
sriedel: unfortunately the spec file still says >= 7.92, upgrade of perl-Mojolicious to >= 8.24 and a restart of the cacheservice would fix it
okurz: yes, I will handle this. Will you make sure to update the spec file?
sriedel: sure
Updated by okurz over 4 years ago
What I did:
sudo salt -l error --state-output=changes '*' cmd.run '/usr/bin/zypper -n patch --with-update --replacefiles --auto-agree-with-licenses --force-resolution --download-in-advance'
sudo salt -l error --state-output=changes '*' cmd.run 'systemctl daemon-reload; systemctl restart openqa-worker-cacheservice openqa-worker-cacheservice-minion'
sudo salt -l error --state-output=changes '*' cmd.run 'rpm -q perl-Mojolicious'
and created https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/232 to correct the auto-update service.
Updated by okurz over 4 years ago
- Related to action #60140: job incompletes failing on initial asset download with "Asset was already requested by another job" added
Updated by nicksinger over 4 years ago
- Is duplicate of action #60629: openqaworker2:21 2:22, 2:23, 2:24 NOT work added
Updated by nicksinger over 4 years ago
- Status changed from In Progress to Rejected
Updated by kraih over 4 years ago
The Mojolicious 8.24 dependency fix has been merged. https://github.com/os-autoinst/openQA/pull/2567
Updated by okurz over 4 years ago
- Due date set to 2019-12-11
- Status changed from Rejected to Feedback
- Priority changed from Urgent to Normal
@nicksinger I think the issue was valid and we should at least keep one of the duplicates. Unless there were more than two?
packages were updated, spec file was updated. Also I merged https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/232 now and would like to keep it open and check myself after some days if the auto-update services behave fine on the workers.
Updated by okurz over 4 years ago
The update service is doing a bit too much now as also os-autoinst and openQA are updated now. Better we do an explicit dup from developed_openQA_Leap
EDIT: 2019-12-05: created https://gitlab.suse.de/openqa/osd-deployment/merge_requests/13
The details of all packages that are about to upgraded are shown in the step "record package versions pre-deploy" of the deployment pipeline, e.g. https://gitlab.suse.de/openqa/osd-deployment/-/jobs/146420/raw
EDIT: 2019-12-06: Merged https://gitlab.suse.de/openqa/osd-deployment/merge_requests/13 and https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/233
Updated by okurz over 4 years ago
- Status changed from Feedback to Resolved
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/149230 shows that also perl-Mojolicious and other packages, not just os-autoinst and such are upgraded. The auto-update service on workers also runs fine and installs only packages. I checked on openqaworker2 and zypper dup today returned no pending upgrades.