Project

General

Profile

Actions

action #125141

open

Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo size:M

Added by mkittler about 1 year ago. Updated about 1 year ago.

Status:
Workable
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
2023-02-28
Due date:
% Done:

0%

Estimated time:

Description

Observation

          ID: security-sensor.repo
    Function: pkgrepo.managed
      Result: False
     Comment: Failed to configure repo 'security-sensor.repo': Zypper command failure: Repository 'security-sensor.repo' is invalid.
              [security-sensor.repo|https://download.opensuse.org/repositories/security:/sensor/15.4] Valid metadata not found at specified URL
              History:
               - Signature verification failed for repomd.xml
               - Can't provide /repodata/repomd.xml

              Please check if the URIs defined for this repository are pointing to a valid repository.
              Skipping repository 'security-sensor.repo' because of the above error.
              Could not refresh the repositories because of errors.Forcing raw metadata refresh
              Retrieving repository 'security-sensor.repo' metadata [..........
              Warning: File 'repomd.xml' from repository 'security-sensor.repo' is unsigned.

                  Note: Signing data enables the recipient to verify that no modifications occurred after the data
                  were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
                  and in extreme cases even to a system compromise.

                  Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
                  whole repo.

                  Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
                  anymore! You should not continue unless you know it's safe.

              File 'repomd.xml' from repository 'security-sensor.repo' is unsigned, continue? [yes/no] (no): no
              error]
     Started: 09:39:50.917365
    Duration: 9775.41 ms
     Changes:   
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: One or more requisite failed: security_sensor.security-sensor.repo
     Started: 09:40:00.699471
    Duration: 0.011 ms
     Changes:
…
Summary for tumblesle
--------------
Succeeded: 231 (changed=1)
Failed:      2
--------------
Total states run:     233

(https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1427053/raw)

Suggestions

  • Find out what the host "tumblesle" is -> a VM on qamaster.qa.suse.de (according to https://racktables.suse.de/index.php?page=object&tab=default&object_id=1300), the full domain is tumblesle.qa.suse.de
  • Check whether the problem persists -> no the repo can be refreshed (on tumblesle)
  • Check whether the error handling (retries) is in accordance with how other repos are configured -> we use pkgrepo.managed: - retry: attempts: 5 for our own devel repos, maybe the same would make sense for security:sensor as well; we don't have a retry for all repos configured via pkgrepo.managed so far, though

Remarks

  • Likely not specific to "tumblesle".
  • Looks like a temporary signing problem of security-sensor.repo (and not like a network issue). DONE So maybe a one-time issue and we don't need to introduce a retry. -> It is reproducible on tumblesle.qa.suse.de with
for i in {001..100}; do echo "## $i" && zypper ref --force -r security-sensor.repo; done

after 23 runs. Directly afterwards it was working to retrieve the file.

  • Optional Try to reproduce the above problem in a clean container environment, at best for crosschecking both Leap and Tumbleweed
  • Based on the above report an issue to zypper on https://github.com/openSUSE/zypper/ as zypper claims "File is unsigned" which is apparently not true. It's likely a temporary connection issue. Better retry
  • Optional: Additionally report an issue with the openSUSE infrastructure with a cross-reference
Actions #1

Updated by mkittler about 1 year ago

  • Subject changed from Salt pillars deployment pipeline failed due to invalid security sensor repo to Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo
  • Description updated (diff)
  • Status changed from New to Feedback
  • Target version set to Ready

I'll keep this in feedback after gathering more info and adding it to the ticket. We can discuss this ticket in the next estimation meeting.

Actions #2

Updated by okurz about 1 year ago

  • Tags set to infra, salt, alert, pillars, tumblesle, reactive work

The problem looks familiar for problems I observed in other scopes regarding signing on download.opensuse.org. I suggest we look for a more generic "upstream solution", e.g. ask around in broader scope chat rooms, mailing lists, etc.

Actions #3

Updated by okurz about 1 year ago

  • Subject changed from Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo to Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo size:M
  • Description updated (diff)
  • Status changed from Feedback to Workable
  • Assignee deleted (mkittler)

Estimated with mkittler and we could also reproduce easily:

for i in {001..100}; do echo "## $i" && zypper ref --force -r security-sensor.repo; done
Actions #4

Updated by okurz about 1 year ago

  • Target version changed from Ready to future
Actions #5

Updated by livdywan about 1 year ago

Another incidence this morning: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1457431/raw

          ID: security-sensor.repo
    Function: pkgrepo.managed
      Result: False
     Comment: Failed to configure repo 'security-sensor.repo': Zypper command failure: Repository 'security-sensor.repo' is invalid.
              [security-sensor.repo|https://download.opensuse.org/repositories/security:/sensor/15.4] Valid metadata not found at specified URL
              History:
               - [|] Error trying to read from 'https://download.opensuse.org/repositories/security:/sensor/15.4'
               - Download (curl) error for 'https://download.opensuse.org/repositories/security:/sensor/15.4/content':
                 Error code: Connection failed
                 Error message: Could not resolve host: download.opensuse.org

              Please check if the URIs defined for this repository are pointing to a valid repository.
              Skipping repository 'security-sensor.repo' because of the above error.
              Could not refresh the repositories because of errors.Forcing raw metadata refresh
              Retrieving repository 'security-sensor.repo' metadata [.error]
     Started: 08:54:20.143484
    Duration: 794.301 ms
     Changes:   
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: One or more requisite failed: security_sensor.security-sensor.repo
     Started: 08:54:20.947657
    Duration: 0.047 ms
     Changes:   
Actions

Also available in: Atom PDF