Project

General

Profile

Actions
Copy link

action #125141

closed

Salt state security-sensor.repo fails regularly due to invalid repo contents from the velociraptor project size:M

Added by mkittler almost 2 years ago. Updated 21 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
2023-02-28
Due date:
% Done:

0%

Estimated time:
Tags:
alert, salt, infra, reactive work, pillars, tumblesle

Description

Observation

          ID: security-sensor.repo
    Function: pkgrepo.managed
      Result: False
     Comment: Failed to configure repo 'security-sensor.repo': Zypper command failure: Repository 'security-sensor.repo' is invalid.
              [security-sensor.repo|https://download.opensuse.org/repositories/security:/sensor/15.4] Valid metadata not found at specified URL
              History:
               - Signature verification failed for repomd.xml
               - Can't provide /repodata/repomd.xml

              Please check if the URIs defined for this repository are pointing to a valid repository.
              Skipping repository 'security-sensor.repo' because of the above error.
              Could not refresh the repositories because of errors.Forcing raw metadata refresh
              Retrieving repository 'security-sensor.repo' metadata [..........
              Warning: File 'repomd.xml' from repository 'security-sensor.repo' is unsigned.

                  Note: Signing data enables the recipient to verify that no modifications occurred after the data
                  were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
                  and in extreme cases even to a system compromise.

                  Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
                  whole repo.

                  Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
                  anymore! You should not continue unless you know it's safe.

              File 'repomd.xml' from repository 'security-sensor.repo' is unsigned, continue? [yes/no] (no): no
              error]
     Started: 09:39:50.917365
    Duration: 9775.41 ms
     Changes:   
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: One or more requisite failed: security_sensor.security-sensor.repo
     Started: 09:40:00.699471
    Duration: 0.011 ms
     Changes:
…
Summary for tumblesle
--------------
Succeeded: 231 (changed=1)
Failed:      2
--------------
Total states run:     233

(https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1427053/raw)

Suggestions

  • Find out what the host "tumblesle" is -> a VM on qamaster.qa.suse.de (according to https://racktables.suse.de/index.php?page=object&tab=default&object_id=1300), the full domain is tumblesle.qa.suse.de
  • Check whether the problem persists -> no the repo can be refreshed (on tumblesle)
  • Check whether the error handling (retries) is in accordance with how other repos are configured -> we use pkgrepo.managed: - retry: attempts: 5 for our own devel repos, maybe the same would make sense for security:sensor as well; we don't have a retry for all repos configured via pkgrepo.managed so far, though

Remarks

  • Likely not specific to "tumblesle".
  • Looks like a temporary signing problem of security-sensor.repo (and not like a network issue). DONE So maybe a one-time issue and we don't need to introduce a retry. -> It is reproducible on tumblesle.qa.suse.de with
for i in {001..100}; do echo "## $i" && zypper ref --force -r security-sensor.repo; done

after 23 runs. Directly afterwards it was working to retrieve the file.

  • Optional Try to reproduce the above problem in a clean container environment, at best for crosschecking both Leap and Tumbleweed
  • Based on the above report an issue to zypper on https://github.com/openSUSE/zypper/ as zypper claims "File is unsigned" which is apparently not true. It's likely a temporary connection issue. Better retry
  • Optional: Additionally report an issue with the openSUSE infrastructure with a cross-reference
Actions
Copy link
 #1

Updated by mkittler almost 2 years ago

  • Subject changed from Salt pillars deployment pipeline failed due to invalid security sensor repo to Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo
  • Description updated (diff)
  • Status changed from New to Feedback
  • Target version set to Ready

I'll keep this in feedback after gathering more info and adding it to the ticket. We can discuss this ticket in the next estimation meeting.

Actions
Copy link
 #2

Updated by okurz almost 2 years ago

  • Tags set to infra, salt, alert, pillars, tumblesle, reactive work

The problem looks familiar for problems I observed in other scopes regarding signing on download.opensuse.org. I suggest we look for a more generic "upstream solution", e.g. ask around in broader scope chat rooms, mailing lists, etc.

Actions
Copy link
 #3

Updated by okurz almost 2 years ago

  • Subject changed from Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo to Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo size:M
  • Description updated (diff)
  • Status changed from Feedback to Workable
  • Assignee deleted (mkittler)

Estimated with mkittler and we could also reproduce easily:

for i in {001..100}; do echo "## $i" && zypper ref --force -r security-sensor.repo; done
Actions
Copy link
 #4

Updated by okurz almost 2 years ago

  • Target version changed from Ready to future
Actions
Copy link
 #5

Updated by livdywan almost 2 years ago

Another incidence this morning: https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1457431/raw

          ID: security-sensor.repo
    Function: pkgrepo.managed
      Result: False
     Comment: Failed to configure repo 'security-sensor.repo': Zypper command failure: Repository 'security-sensor.repo' is invalid.
              [security-sensor.repo|https://download.opensuse.org/repositories/security:/sensor/15.4] Valid metadata not found at specified URL
              History:
               - [|] Error trying to read from 'https://download.opensuse.org/repositories/security:/sensor/15.4'
               - Download (curl) error for 'https://download.opensuse.org/repositories/security:/sensor/15.4/content':
                 Error code: Connection failed
                 Error message: Could not resolve host: download.opensuse.org

              Please check if the URIs defined for this repository are pointing to a valid repository.
              Skipping repository 'security-sensor.repo' because of the above error.
              Could not refresh the repositories because of errors.Forcing raw metadata refresh
              Retrieving repository 'security-sensor.repo' metadata [.error]
     Started: 08:54:20.143484
    Duration: 794.301 ms
     Changes:   
----------
          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: One or more requisite failed: security_sensor.security-sensor.repo
     Started: 08:54:20.947657
    Duration: 0.047 ms
     Changes:
Actions
Copy link
 #7

Updated by nicksinger 3 months ago

  • Subject changed from Salt pillars deployment pipeline failed on "tumblesle" due to invalid security sensor repo size:M to Salt state security-sensor.repo fails regularly due to invalid repo contents from the velociraptor project size:M

Similar case on OSD today while doing a manual deployment:

          ID: security-sensor.repo
    Function: pkg.latest
        Name: velociraptor-client
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/salt/state.py", line 2402, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1282, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/states/pkg.py", line 2659, in latest
                  *desired_pkgs, fromrepo=fromrepo, refresh=refresh, **kwargs
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1234, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
                  return callable(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 828, in latest_version
                  package_info = info_available(*names, **kwargs)
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 752, in info_available
                  "info", "-t", "package", *batch[:batch_size]
                File "/usr/lib/python3.6/site-packages/salt/modules/zypperpkg.py", line 439, in __call
                  salt.utils.stringutils.to_str(self.__call_result["stdout"])
                File "/usr/lib64/python3.6/xml/dom/minidom.py", line 1968, in parseString
                  return expatbuilder.parseString(string)
                File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 925, in parseString
                  return builder.parseString(string)
                File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 223, in parseString
                  parser.Parse(string, True)
              xml.parsers.expat.ExpatError: syntax error: line 1, column 0
     Started: 08:37:38.097036
    Duration: 3734.262 ms
     Changes:
Actions
Copy link
 #8

Updated by okurz about 1 month ago

  • Parent task set to #159324
Actions
Copy link
 #9

Updated by okurz 21 days ago

  • Category set to Regressions/Crashes
  • Status changed from Workable to Resolved
  • Assignee set to okurz
  • Target version changed from future to Ready

that problem did not appear anymore since we use an IBS repo or just rely on official openSUSE repositories.

Actions
Copy link

Also available in: Atom PDF