Project

General

Profile

action #123867

[sporadic][ci] circleCI job "build-docs-nightly" failed

Added by okurz about 2 months ago. Updated 17 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
tinita
Category:
Concrete Bugs
Target version:
Ready
Start date:
2023-02-01
Due date:
% Done:

0%

Estimated time:
Difficulty:
Tags:
reactive work

Description

Observation

https://app.circleci.com/jobs/github/os-autoinst/openQA/102194 from night 2023-02-01 failed with:

#!/bin/bash -eo pipefail
tools/ci/build_or_install_from_cache.sh

+ '[' -z 857c64fd-5f56-40c7-a8da-8e548e2598c5 ']'
++ find /var/cache/zypp/packages/
++ grep '.rpm$'
+ packages=
+ echo 'No RPM packages found, cache is empty, aborting'
No RPM packages found, cache is empty, aborting
+ exit 1

Reproducible

Happens consistently since 2023-01-27, first occurence https://app.circleci.com/pipelines/github/os-autoinst/openQA/10916/workflows/be95b01f-7138-4e53-b13c-c9296c5744c2/jobs/102185, see https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master

Acceptance criteria

  • AC1: CI job "build-docs-nightly" consistently passes
  • AC2: CI badge on github.com/os-autoinst/openQA/ is green

Related issues

Related to openQA Project - action #123661: Use non-personal or in-team tokens for openQA OBS CI integration size:MResolved2023-01-25

History

#1 Updated by okurz about 2 months ago

  • Description updated (diff)

#2 Updated by okurz about 2 months ago

  • Related to action #123661: Use non-personal or in-team tokens for openQA OBS CI integration size:M added

#3 Updated by okurz about 2 months ago

Could this be related to #123661?

#4 Updated by okurz about 2 months ago

  • Priority changed from High to Urgent

#5 Updated by cdywan about 2 months ago

  • Status changed from New to In Progress
  • Assignee set to cdywan

The nightly pipeline says No cache is found for key: v1-owmJYn8136qZKFx1O2O8UoqbTsBzd0DjM2YjZqqLN_E=-hFwLTWZid4qDmmr9j3q4rAJ_BQVBgBI6DFnUL1mz4Z8= whilst the last test pipeline claims Skipping cache generation, cache already exists for key: v1-iPD3r2kpvBVaIoEv7hzbxNj39g9NvPA15c9Ad+974Rw=-bxOu+vZwCVc3zx37f80NbWHLZzNLIJcn0Rjcr84R1G8=.

This isn't the same key.

I'll try and re-trigger it manually. Maybe it was simply a race? I don't actually see it failing to generate a cache.

#6 Updated by okurz about 2 months ago

I don't think retriggering can help for an issue that reproduces for days. Or do you retrigger something else?

#7 Updated by cdywan about 2 months ago

  • Status changed from In Progress to Feedback

okurz wrote:

I don't think retriggering can help for an issue that reproduces for days. Or do you retrigger something else?

After re-triggering cache.fullstack and also build docs nightly both jobs finished successfully.

I assume we hit the default/ maximum cache expiry of 15 days.

#8 Updated by cdywan about 2 months ago

There isn't really a good way to avoid the race completely, but what we can do is make the error message more helpful: https://github.com/os-autoinst/openQA/pull/4992

#9 Updated by cdywan about 2 months ago

  • Status changed from Feedback to Resolved

The PR has been merged. I've seen at least one nightly job that was automatically triggered and finished successfully, so I assume we're good here.

#10 Updated by tinita 19 days ago

Maybe adding the cache.fullstack job to the nightly would help?

#11 Updated by tinita 19 days ago

  • Status changed from Resolved to In Progress
  • Assignee changed from cdywan to tinita
  • Priority changed from Urgent to Normal

#12 Updated by tinita 19 days ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openQA/pull/5027 Run cache.fullstack job also in nightly job

#13 Updated by tinita 17 days ago

  • Status changed from Feedback to Resolved

The latest nightly ran fine.
Resolving. If it pops up again we will notice.

Also available in: Atom PDF