Project

General

Profile

Actions

action #123867

closed

[sporadic][ci] circleCI job "build-docs-nightly" failed

Added by okurz almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-02-01
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://app.circleci.com/jobs/github/os-autoinst/openQA/102194 from night 2023-02-01 failed with:

#!/bin/bash -eo pipefail
tools/ci/build_or_install_from_cache.sh

+ '[' -z 857c64fd-5f56-40c7-a8da-8e548e2598c5 ']'
++ find /var/cache/zypp/packages/
++ grep '.rpm$'
+ packages=
+ echo 'No RPM packages found, cache is empty, aborting'
No RPM packages found, cache is empty, aborting
+ exit 1

Reproducible

Happens consistently since 2023-01-27, first occurence https://app.circleci.com/pipelines/github/os-autoinst/openQA/10916/workflows/be95b01f-7138-4e53-b13c-c9296c5744c2/jobs/102185, see https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master

Acceptance criteria

  • AC1: CI job "build-docs-nightly" consistently passes
  • AC2: CI badge on github.com/os-autoinst/openQA/ is green

Related issues 1 (0 open1 closed)

Related to openQA Project - action #123661: Use non-personal or in-team tokens for openQA OBS CI integration size:MResolvedokurz2023-01-25

Actions
Actions #1

Updated by okurz almost 2 years ago

  • Description updated (diff)
Actions #2

Updated by okurz almost 2 years ago

  • Related to action #123661: Use non-personal or in-team tokens for openQA OBS CI integration size:M added
Actions #3

Updated by okurz almost 2 years ago

Could this be related to #123661?

Actions #4

Updated by okurz almost 2 years ago

  • Priority changed from High to Urgent
Actions #5

Updated by livdywan almost 2 years ago

  • Status changed from New to In Progress
  • Assignee set to livdywan

The nightly pipeline says No cache is found for key: v1-owmJYn8136qZKFx1O2O8UoqbTsBzd0DjM2YjZqqLN_E=-hFwLTWZid4qDmmr9j3q4rAJ_BQVBgBI6DFnUL1mz4Z8= whilst the last test pipeline claims Skipping cache generation, cache already exists for key: v1-iPD3r2kpvBVaIoEv7hzbxNj39g9NvPA15c9Ad+974Rw=-bxOu+vZwCVc3zx37f80NbWHLZzNLIJcn0Rjcr84R1G8=.

This isn't the same key.

I'll try and re-trigger it manually. Maybe it was simply a race? I don't actually see it failing to generate a cache.

Actions #6

Updated by okurz almost 2 years ago

I don't think retriggering can help for an issue that reproduces for days. Or do you retrigger something else?

Actions #7

Updated by livdywan almost 2 years ago

  • Status changed from In Progress to Feedback

okurz wrote:

I don't think retriggering can help for an issue that reproduces for days. Or do you retrigger something else?

After re-triggering cache.fullstack and also build docs nightly both jobs finished successfully.

I assume we hit the default/ maximum cache expiry of 15 days.

Actions #8

Updated by livdywan almost 2 years ago

There isn't really a good way to avoid the race completely, but what we can do is make the error message more helpful: https://github.com/os-autoinst/openQA/pull/4992

Actions #9

Updated by livdywan almost 2 years ago

  • Status changed from Feedback to Resolved

The PR has been merged. I've seen at least one nightly job that was automatically triggered and finished successfully, so I assume we're good here.

Actions #10

Updated by tinita over 1 year ago

Maybe adding the cache.fullstack job to the nightly would help?

Actions #11

Updated by tinita over 1 year ago

  • Status changed from Resolved to In Progress
  • Assignee changed from livdywan to tinita
  • Priority changed from Urgent to Normal
Actions #12

Updated by tinita over 1 year ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openQA/pull/5027 Run cache.fullstack job also in nightly job

Actions #13

Updated by tinita over 1 year ago

  • Status changed from Feedback to Resolved

The latest nightly ran fine.
Resolving. If it pops up again we will notice.

Actions #14

Updated by tinita 15 days ago

  • Copied to action #169246: [ci] circleCI job "build-docs-nightly" failed size:S added
Actions #15

Updated by tinita 15 days ago

  • Copied to deleted (action #169246: [ci] circleCI job "build-docs-nightly" failed size:S)
Actions

Also available in: Atom PDF