action #60626

"Download .* processed" but ending up with setup-failure, workers are missing updated perl-Mojolicious >= 8.24

Added by okurz 7 months ago. Updated 7 months ago.

Concrete Bugs
Target version:
Start date:
Due date:
% Done:


Estimated time:
Duration: 6


Since today's deployment on osd we have many incompletes like

[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] +++ setup notes +++
[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] Start time: 2019-12-04 10:24:09
[2019-12-04T11:24:09.0248 CET] [info] [pid:25781] Running on malbec:2 (Linux 4.12.14-lp151.27-default #1 SMP Fri May 10 14:13:15 UTC 2019 (862c838) ppc64le)
[2019-12-04T11:24:09.0256 CET] [debug] [pid:25781] Downloading sle-12-SP2-ppc64le-4.4.121-190.1.gd3cd4b0-Server-DVD-Incidents-Kernel@ppc64le-virtio-with-ltp.qcow2, request #43477 sent to Cache Service
[2019-12-04T11:24:14.0320 CET] [debug] [pid:25781] Download of sle-12-SP2-ppc64le-4.4.121-190.1.gd3cd4b0-Server-DVD-Incidents-Kernel@ppc64le-virtio-with-ltp.qcow2 processed
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] +++ worker notes +++
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] End time: 2019-12-04 10:24:14
[2019-12-04T11:24:14.0352 CET] [info] [pid:25781] Result: setup failure
[2019-12-04T11:24:14.0355 CET] [info] [pid:48327] Uploading autoinst-log.txt

triggered most likely by one of the following changes in openQA-worker:

  • Increase SQLite busy timeout and reduce the number of concurrent SQLite connections
  • Show log content in test details tab on no module results
  • Fix asset definition of SVG icon for running jobs
  • Check generate-documentation if target branch exists
  • Calculate coverage explicitly in Selenium tests
  • Improve group property HTML template
  • travis_test: Add requirements for spellcheck in os-autoinst
  • Use the cache tmp directory for temporary download files created by the cache service
  • Use info instead of warn messages (so monitoring does not get triggered) and quote all string values
  • Allow for very quiet cache service tests
  • Be a little less verbose with init log messages
  • Allow from_worker to set attributes as well
  • Remove redundant error message
  • Format cache service logs to start with a newline
  • Improve cache service log messages to help with debugging issues of missing assets
  • Log all cache service output on both sides (cache service and worker) and fix many small test issues
  • Handle cache model logging through an attribute


sriedel: Got it, the Mojolicious version is somehow not high enough on openqaworker6
okurz: all osd workers
sriedel: it's probably higher on O3, that's why everything worked fine there we need perl-Mojolicious 8.24
okurz: we have the same on all osd and it's 8.18, on o3 workers it's 8.26
sriedel: unfortunately the spec file still says >= 7.92, upgrade of perl-Mojolicious to >= 8.24 and a restart of the cacheservice would fix it
okurz: yes, I will handle this. Will you make sure to update the spec file?
sriedel: sure

Related issues

Related to openQA Project - action #60140: job incompletes failing on initial asset download with "Asset was already requested by another job"Resolved2019-01-28

Is duplicate of openQA Infrastructure - action #60629: openqaworker2:21 2:22, 2:23, 2:24 NOT workRejected2019-12-04


#1 Updated by okurz 7 months ago

What I did:

sudo salt -l error --state-output=changes '*' '/usr/bin/zypper -n patch --with-update --replacefiles --auto-agree-with-licenses --force-resolution --download-in-advance'
sudo salt -l error --state-output=changes '*' 'systemctl daemon-reload; systemctl restart openqa-worker-cacheservice openqa-worker-cacheservice-minion'
sudo salt -l error --state-output=changes '*' 'rpm -q perl-Mojolicious'

and created to correct the auto-update service.

#2 Updated by okurz 7 months ago

  • Related to action #60140: job incompletes failing on initial asset download with "Asset was already requested by another job" added

#3 Updated by nicksinger 7 months ago

  • Is duplicate of action #60629: openqaworker2:21 2:22, 2:23, 2:24 NOT work added

#4 Updated by nicksinger 7 months ago

  • Status changed from In Progress to Rejected

#5 Updated by kraih 7 months ago

The Mojolicious 8.24 dependency fix has been merged.

#6 Updated by okurz 7 months ago

  • Due date set to 2019-12-11
  • Status changed from Rejected to Feedback
  • Priority changed from Urgent to Normal

nicksinger I think the issue was valid and we should at least keep one of the duplicates. Unless there were more than two?

packages were updated, spec file was updated. Also I merged now and would like to keep it open and check myself after some days if the auto-update services behave fine on the workers.

#7 Updated by okurz 7 months ago

The update service is doing a bit too much now as also os-autoinst and openQA are updated now. Better we do an explicit dup from developed_openQA_Leap

EDIT: 2019-12-05: created

The details of all packages that are about to upgraded are shown in the step "record package versions pre-deploy" of the deployment pipeline, e.g.

EDIT: 2019-12-06: Merged and

#8 Updated by okurz 7 months ago

  • Status changed from Feedback to Resolved shows that also perl-Mojolicious and other packages, not just os-autoinst and such are upgraded. The auto-update service on workers also runs fine and installs only packages. I checked on openqaworker2 and zypper dup today returned no pending upgrades.

Also available in: Atom PDF