action #169159
closedAllow variable expansion incorporating worker settings size:S
0%
Description
Motivation¶
This would be useful so job settings referring to assets (e.g. QA_HEAD_REPO=https://download.suse.de/ibs/…) could be written in a more generic way (e.g. QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/…). This way the download server could be replaced by a mirror more easily. Considering ideas like https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 it would be useful if worker settings are considered as well. See https://suse.slack.com/archives/C02CANHLANP/p1730383116044599 for the related discussion with @szarate.
This allows convenient variable evaluation in the worker so that we do not need to do that in test code, e.g. osado, in various place.
Acceptance criteria¶
- AC1: Placeholders in job settings are substituted taking settings from the worker config into account.
- AC2: The variable substitution introduced via AC1 is in-line with https://open.qa/docs/#_variable_expansion (and internally uses the same code as much as possible).
Suggestions¶
- Check what function is used to implement placeholders. Invoke that function in
isotovideo.pmin the openQA-worker as well before the asset handling is done. - The webUI already replaces variables. We could keep variables
- Consider a default value, e.g.
%REPO_MIRROR_HOST:my.default.host%so that we don't need to set variables on all workers. The default must then only be evaluated in the Last Instance (!) wherever that is, possibly worker. As alternative os-autoinst can be used.
Further details¶
- This feature is intended e.g. to be used with SLE maintenance tests so also qem-bot would need to be changed to only set a variable, possibly with default value
- For the purpose of OSD this feature only is usable if non-CC OSD workers have access to openqa.suse.de webUI, e.g. need other currently open tickets from https://progress.opensuse.org/issues?parent_id=~165282&set_filter=true&status_id=o
- The initial part about "before the worker cache asset download" does not make sense because worker asset caching only downloads assets from OSD, never from external resources so variable expansion is not useful there
Updated by szarate 2 months ago
- Related to action #168097: [qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de added
Updated by okurz 2 months ago
- Assignee set to okurz
- Parent task set to #166598
mkittler wrote:
- AC1: Placeholders in job settings are substituted before the asset download taking settings from the worker config into account.
if that would be true how would worker specific settings be evaluated then?
Also, can you please explain the relation to already existing variable expansion as documented on https://open.qa/docs/#_variable_expansion ?
Updated by szarate 2 months ago
Please read the slack thread mentioned above.
okurz wrote in #note-3:
mkittler wrote:
- AC1: Placeholders in job settings are substituted before the asset download taking settings from the worker config into account.
if that would be true how would worker specific settings be evaluated then?
once the job is assigned to the worker, but before asset downloading starts, if that's not possible an alternative solution is needed
Also, can you please explain the relation to already existing variable expansion as documented on https://open.qa/docs/#_variable_expansion ?
Variable expansion is only supported at job creation time. See slack thread.
In general, we're looking to solve the problem described in the thread, at medium and test setting level.
Updated by okurz 2 months ago
- Subject changed from Allow variable expansion before a job starts to Allow variable expansion incorporating worker settings size:S
- Description updated (diff)
- Status changed from New to Workable
- Assignee deleted (
okurz) - Priority changed from Normal to Urgent
ok, so we clarified some points. I understood that szarate wants to urgently have that so let's give him that :)
@szarate please be aware that this will not help us to get tests running in NUE2 as currently not even NUE2 based OSD workers can connect to OSD webUI which would need the tickets from https://progress.opensuse.org/issues?parent_id=~166598&set_filter=true&status_id=o as well, at time of writing 7, e.g.
Updated by openqa_review 2 months ago
- Due date set to 2024-11-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler 2 months ago
- Status changed from In Progress to Feedback
The change has been deployed. So we could move https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 forward (although I'm currently checking whether we can do better than mentioning the default download.suse.de on all other workers).
Updated by szarate 2 months ago
Marius, I had
mkittler wrote in #note-12:
The change has been deployed. So we could move https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 forward (although I'm currently checking whether we can do better than mentioning the default
download.suse.deon all other workers).
I created: https://progress.opensuse.org/issues/169492 and crossed the respective line on Confluence
Updated by mkittler 2 months ago
- Blocked by action #169564: Configure wireguard tunnels on OSD production hosts needed for openQA located in the NUE2 server room size:S added
Updated by livdywan about 2 months ago
Blocker being unblocked, see #169564#note-17
Updated by mkittler about 2 months ago
- Blocked by deleted (action #169564: Configure wireguard tunnels on OSD production hosts needed for openQA located in the NUE2 server room size:S)
Updated by mkittler about 2 months ago
- Status changed from Blocked to Feedback
@szarate You can now try out this feature as relevant workers can now connect to openQA via Wireguard. For qemu-based jobs you of course don't need this feature and for baremetal tests additional problems still need to be solved (see #168097#note-29). If you can't find a situation where this feature can be used right now I'd nevertheless consider this ticket resolved as the feature is tested via unit tests and I also tested it locally.
Updated by okurz about 2 months ago
- Status changed from Feedback to Resolved
Discussed with szarate. We are happy with the current state.