action #152092
closedopenQA Project (public) - coordination #112862: [saga][epic] Future ideas for easy multi-machine handling: MM-tests as first-class citizens
openQA Project (public) - coordination #111929: [epic] Stable multi-machine tests covering multiple physical workers
Handle all package downgrades in OSD infrastructure properly in salt size:M
0%
Description
Motivation¶
See lessons learned meeting #139136. We just have to accept that there are situations where we need to run downgraded packages as OS bug workarounds but maybe our process for that is lacking. Unless we want to use SUSE Manager likely we should put all those downgrade handling in salt as we already do/did for other cases. So check what we currently have for handling in salt and extend that and document, reference in our process documentation
Acceptance criteria¶
- AC1: All current package locks on OSD salt machines are ensured to be applied on machines by salt
- AC2: Our process documentation includes instructions how to handle downgraded packages
- AC3: It is still fine if we temporarily downgrade some packages, e.g. for bisecting or testing for some hours/days
Suggestions¶
- First see #150965 and https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1065 so if we apply zypper locks with the right comment then updating the system should still work
- Also potentially consider to bring back https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/984 in a general way
- Extend our salt recipes to track a list of downgraded packages and reasons, e.g.
- a list in salt pillars, applicable for certain roles, e.g. "all ppc64le osd openQA workers"
- according handling in salt states to install the specified downgraded package versions
- according zypper lock to prevent upgrades again with according zypper lock comment
- Extend our process documentation includes instructions how to handle downgraded packages, like simply write in the https://gitlab.suse.de/openqa/salt-states-openqa README how to downgrade and apply package locks properly
Updated by okurz about 1 year ago
- Copied from action #139136: Conduct "lessons learned" with Five Why analysis for "test fails in iscsi_client due to salt 'host'/'nodename' confusion" size:M added
Updated by okurz about 1 year ago
- Copied to action #152095: [spike solution][timeboxed:8h] Ping over GRE tunnels and TAP devices and openvswitch outside a VM with differing packet sizes size:S added
Updated by okurz about 1 year ago
- Related to action #150965: At least diesel+petrol+mania fail to auto-update due to kernel locks preventing patches size:M added
Updated by nicksinger 10 months ago
- Status changed from Workable to In Progress
I checked our currently applied locks on all machines with salt '*' cmd.run "zypper ll"
and all of them seem to be valid. In addition to that I understood that https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1065/diffs introduced automatic handling of already applied locks and adds eventually additional locks on e.g. specific patches. This ensures a system can be updated with only a simple lock added manually. So what I'm now going to implement is a simple way of handling these "initial locks" in salt.
Updated by okurz 10 months ago
- Related to action #116812: [qe-core] Leap 15.5 uefi console switch fail size:M added
Updated by openqa_review 10 months ago
- Due date set to 2024-02-29
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger 10 months ago
AC1 and AC3 should be covered by https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/718 and https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1111 - has to be tested if merged. Also not sure where to add AC3 - maybe I just add a README to the newly created "system/packages" sub-directory in our salt states
Updated by okurz 10 months ago
nicksinger wrote in #note-10:
Also not sure where to add AC3 - maybe I just add a README to the newly created "system/packages" sub-directory in our salt states
I suggest instead to just extend the README in the top directory. READMEs in subdirectories are often not found and forgotten.
Updated by nicksinger 10 months ago
- Status changed from In Progress to Feedback
README.md updated with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1113 - I consider all ACs met after this is merged.
Updated by okurz 10 months ago
- Due date deleted (
2024-02-29) - Status changed from Feedback to Resolved
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1113 merged. I agree that this covers everything.
Updated by livdywan 10 months ago
https://gitlab.suse.de/openqa/salt-states-openqa#add-salt-manage-package-locks just for clarity on where the bits can be found now