Project

General

Profile

Actions
Copy link

action #178564

open

Unit tests in salt pipelines failing due to augeas.change and crypt / bsd_shadow size:S

Added by livdywan about 1 month ago. Updated about 1 month ago.

Status:
Blocked
Priority:
Low
Assignee:
nicksinger
Category:
Feature requests
Target version:
QA (public) - future
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:
salt, infra, reactive work

Description

Observation

Current salt pipelines are failing. This wasn't seen as part of alert handling (yet) but very recent individual runs in MR's and re-runs on master.

File "/usr/lib/python3.13/site-packages/salt/modules/bsd_shadow.py", line 22, in <module>
    import salt.utils.pycrypto
  File "/usr/lib/python3.13/site-packages/salt/utils/pycrypto.py", line 104, in <module>
    methods = {m.name.lower(): m for m in crypt.methods}
                                          ^^^^^^^^^^^^^
AttributeError: module 'crypt' has no attribute 'methods'
[...]
          ID: kdump-conf
    Function: augeas.change
      Result: False
     Comment: State 'augeas.change' was not found in SLS 'kdump'
              Reason: 'augeas' __virtual__ returned False: augeas module could not be loaded

Acceptance criteria

  • AC1: Unit tests in salt pipelines are passing

Suggestions

Actions
Copy link
 #1

Updated by livdywan about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by livdywan about 1 month ago

  • Priority changed from Normal to High

I put this in High and Ready because it seems to affect all new pipeline runs

Actions
Copy link
 #3

Updated by livdywan about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by tinita about 1 month ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by nicksinger about 1 month ago

  • Status changed from New to In Progress
  • Assignee set to nicksinger

while trying to move https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1404 forward, I found why this fails. podman run --rm -it registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:salt-minion-git-core-ssh with:

6bdc1d287a87:/srv/salt # tree .
.
|-- test
|   `-- init.sls
`-- top.sls

2 directories, 2 files
6bdc1d287a87:/srv/salt # cat top.sls
base:
  '*':
    - test
6bdc1d287a87:/srv/salt # cat test/init.sls
kdump-conf:
  augeas.change:
    - lens: Shellvars.lns
    - context: /files/etc/sysconfig/kdump
    - changes:
      - set KDUMP_SMTP_SERVER '"relay.suse.de"'
      - set KDUMP_NOTIFICATION_TO '"osd-admins@suse.de"'

and salt-call --local -l debug state.apply mock=True was a good, minimal reproducer. Salt complains about the missing cryptography-module which seems unrelated at first (because the x509-module shows other deprecation-warnings as well) but it turns out that zypper in python311-cryptography already fixes the issue. Checking the container definition at https://build.opensuse.org/projects/home:okurz:container:ca/packages/salt-minion-git-core-ssh/files/Dockerfile?expand=1 I found that python3-setuptools python3-augeas python3-cryptography are all not available any longer in the repository and seem be replaced by the explicit 311-packages. Not sure how to approach the fix without being too specific.

Actions
Copy link
 #6

Updated by livdywan about 1 month ago

  • Subject changed from Unit tests in salt pipelines failing due to augeas.change and crypt / bsd_shadow to Unit tests in salt pipelines failing due to augeas.change and crypt / bsd_shadow size:S
  • Description updated (diff)
Actions
Copy link
 #7

Updated by openqa_review about 1 month ago

  • Due date set to 2025-03-26

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #8

Updated by nicksinger about 1 month ago

So I tried to adjust our container to install salt-minion with zypper -n in --recommends salt-minion in https://build.opensuse.org/package/rdiff/home:nicksinger:branches:home:okurz:container:ca/salt-minion-git-core-ssh?linkrev=base&rev=2
It took me a while to understand OBS correctly but eventually tried my minimal reproducer (from https://progress.opensuse.org/issues/178564#note-5) locally with podman run --rm -it registry.opensuse.org/home/nicksinger/branches/home/okurz/container/ca/containers/tumbleweed:salt-minion-git-core-ssh. This fixes the need to explicitly mention python3-setuptools python3-cryptography (which would pull in the python313 version anyway) but misses out on python3-augeas. I think this is a packaging-bug (as salt-minion should suggest the correct python311-augeas package). I had a look at https://build.opensuse.org/projects/openSUSE:Factory/packages/salt/files/salt.spec?expand=1 but it is rather complex so I started with asking in #discuss-salt first: https://suse.slack.com/archives/C02JMF41G9E/p1741791334976679

Actions
Copy link
 #9

Updated by nicksinger about 1 month ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #10

Updated by nicksinger about 1 month ago

  • Status changed from Feedback to In Progress

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1403 - now the stage "test" fails because the python module "spwd" seems to be outdated: https://github.com/saltstack/salt/issues/64264

Actions
Copy link
 #11

Updated by nicksinger about 1 month ago

  • Status changed from In Progress to Feedback
  • Priority changed from High to Normal

https://build.opensuse.org/request/show/1254067 to downgrade even further to 3.11

Actions
Copy link
 #12

Updated by okurz about 1 month ago

  • Status changed from Feedback to Workable
  • Priority changed from Normal to High

https://build.opensuse.org/requests/1254067 accepted.

I triggered jobs as part of a new pipeline with jobs which look like they fail probably different now, e.g.
https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/4001294#L948
says

ModuleNotFoundError: No module named 'cryptography'

but now executed with the python3.11 stack.

@nicksinger you reduced from "High" to "Normal". Did I miss a workaround that mitigates that? I triggered a new pipeline so I assume that the right image and current code is picked up but maybe you expected something different?

Actions
Copy link
 #13

Updated by okurz about 1 month ago

  • Status changed from Workable to In Progress

accepted new SR https://build.opensuse.org/requests/1254365. Not yet published at time of writing.

Actions
Copy link
 #14

Updated by okurz about 1 month ago

  • Due date deleted (2025-03-26)
  • Status changed from In Progress to Blocked
  • Target version changed from Ready to future

Published and seems to help. We retriggered (and rebased and retriggered) multiple MRs. E.g. https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines shows stable.
Let's block on https://github.com/openSUSE/salt-packaging/pull/108

Actions
Copy link
 #15

Updated by okurz about 1 month ago

  • Priority changed from High to Low
Actions
Copy link

Also available in: Atom PDF