Project

General

Profile

action #38807

Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used

Added by okurz about 3 years ago. Updated 6 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2018-07-25
Due date:
% Done:

0%

Estimated time:
Difficulty:
medium

Description

Motivation

We use the medium "iso/SLE-12-SP4-Server-DVD-x86_64-Build????-Media2.iso" while using it as repo.
We thought we used that ISO to extract the repository, but the ISO and the repository are synced and handle into OSD in different tasks, so we don't need to sync nor have a job setting with those unused isos.

However we need to account for the medium (so it will get deleted by openqa-gru.service), see #38804 for fix to medium2.
There are still some other ISOs for modules that need to be checked.

Acceptance criteria

  • AC1: All synced assets are accounted for in test asset variables
  • AC2: No unused assets are provided as assets on jobs that users want to clone
  • AC3: Jobs without ISO do not have the setting ISO_MAXSIZE

Suggestions

  • For AC1 check https://openqa.suse.de/admin/assets for "Untracked assets" to correspond with QSF-u owned tests. If there are any make sure the tests reference the assets, e.g. by using ISO_1, HDD_1, etc. to reference these assets.
  • For AC2 ensure no test suites or media or job templates reference assets which tests do not need, e.g. no s390x tests need an ISO. As ISO is often provided by rsync.pl, overwrite assets with e.g. +ISO= or +ISO_1= or similar. Also make sure if assets are not needed that they are not synced by rsync.pl nor ObsRsync
  • For AC3 simply follow the previous step and make sure that ISO_MAXSIZE is only specified on tests that also actually use the ISO assets

Further details

s390x

Our current s390x jobs reference an ISO as well but we never use the ISO directly but only repos from that. These jobs should also be handled


Related issues

Related to openQA Tests - action #38804: [functional][u][fast] Make sure Media2 isos from SLE12SP4 are accounted for in ISO_* variables in any jobResolved2018-07-252018-07-31

Related to openQA Tests - action #59394: [qe-core][functional] Overwrite empty ISO variable everywhere where not needed, i.e. `+ISO=`, to prevent useless ISO downloading and storageBlocked2019-11-13

History

#1 Updated by okurz about 3 years ago

Maybe we can simply delete the ISO again during execution of rsync.pl after extracting repos when we do not actually use the ISO. Would be an option if we have rsync.pl abort with an error when it could not trigger any tests using an asset at all.

#2 Updated by okurz about 3 years ago

  • Due date set to 2018-08-14
  • Target version changed from Milestone 19 to Milestone 18

With the current approach taken in #38804 it was identified that we do not use the Media2.iso anywhere so maybe we can apply the same approach for e.g. the s390x tests which do not actually use the ISO?

SLindoMansilla if this is something for you we can work on it pulling it into the current sprint or in next sprint.

#3 Updated by SLindoMansilla about 3 years ago

  • Related to action #38804: [functional][u][fast] Make sure Media2 isos from SLE12SP4 are accounted for in ISO_* variables in any job added

#4 Updated by SLindoMansilla about 3 years ago

  • Description updated (diff)
  • Status changed from Workable to In Progress
  • Assignee set to SLindoMansilla

I would let it on the next sprint, since it was not planned for the current sprint nor is an urgent issue (or is it? then we could [fast] it)

#5 Updated by SLindoMansilla about 3 years ago

  • Description updated (diff)

#6 Updated by SLindoMansilla about 3 years ago

  • Description updated (diff)
  • Status changed from In Progress to Feedback

I have found some places in code where the ISO is used:

    elsif (get_var('BUILD_WE') && get_var('ISO_2')) {
        $ar_url = 'dvd:///?devices=/dev/sr2';
            send_key_until_needlematch 'addon-dvd-list',         'tab',  5;     # jump into addon list
            send_key_until_needlematch "addon-dvd-sr$sr_number", 'down', 10;    # select addon in list

In the case of any test suite for ADDONS=(RT, SDK, HA, GEO, WE, etc)

WE

#7 Updated by SLindoMansilla about 3 years ago

  • Status changed from Feedback to In Progress

The setting has to come from https://gitlab.suse.de/SLindoMansilla/scripts/blob/master/rsync.pl

#8 Updated by SLindoMansilla about 3 years ago

MR: https://gitlab.suse.de/openqa/scripts/merge_requests/240

Merged. Waiting for verification run on OSD.

#9 Updated by SLindoMansilla about 3 years ago

  • Status changed from In Progress to Feedback

#10 Updated by SLindoMansilla about 3 years ago

  • Status changed from Feedback to In Progress

We can continue looking for more unused ISOs.

#11 Updated by SLindoMansilla about 3 years ago

IPMI also doesn't use the ISO

#12 Updated by SLindoMansilla about 3 years ago

rsync.pl

rsync.pl just post the ISO taking care of the architecture, but not the type of machine, IPMI is x86_64.
We need the ISO for x86_64 non-IPMI architectures (qemu, no real DVD-Player).
But we can deactivate the ISO setting for IPMI so it is not registered as a job asset, not needed to have it to run the job, and not needed to download it when cloning an IPMI job.

The ISO setting set by rsync.pl can be removed for all x86_64, but then the ISO should be at least set on Medium, Machines or Test Suites for non-IPMI on the openQA webui.

openQA webui settings

There is no Medium for IPMI, the medium is for x86_64.

Test suites are also general, so it is not possible to deactivate the setting 'ISO=' only for IPMI from there.

A solution could be to deactivate it 'ISO=' on the IPMI machines. But the overriding precedence is

  1. Medium: https://openqa.suse.de/admin/products
  2. Machine: https://openqa.suse.de/admin/machines
  3. Test Suite: https://openqa.suse.de/admin/test_suites

So, Test Suites that set the ISO would override the settings even for IPMI Machines. It should be possible to give higher priority to the Machine settings by preceding the setting with '+'.

For example for Machine 64bit-ipmi:

+ISO=
SERIALDEV=ttyS1
TIMEOUT_SCALE=3
VNC_TYPING_LIMIT=5
WORKER_CLASS=64bit-ipmi
_CHKSEL_RATE_WAIT_TIME=120
comment=poo#18514 about the VNC typing limit

#13 Updated by okurz about 3 years ago

  • Due date changed from 2018-08-14 to 2018-08-28

bulk move to next sprint as could not be discussed in SR

#14 Updated by SLindoMansilla about 3 years ago

  • Status changed from In Progress to Workable

I need to keep investigating on this ticket. But, at the moment I am working on ticket that needs less time to be resolved.

#15 Updated by SLindoMansilla about 3 years ago

  • Subject changed from [functional][u] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to [functional][u][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Estimated time set to 8.00 h
  • Difficulty set to medium

#16 Updated by mgriessmeier about 3 years ago

  • Due date changed from 2018-08-28 to 2018-09-11

move due to vacation

#17 Updated by SLindoMansilla about 3 years ago

I haven't work on this during this sprint.

#18 Updated by mgriessmeier about 3 years ago

  • Due date changed from 2018-09-11 to 2018-09-25

#19 Updated by okurz about 3 years ago

are you really working on it?

#20 Updated by okurz about 3 years ago

  • Assignee deleted (SLindoMansilla)
  • Target version changed from Milestone 18 to Milestone 19

guess not …

#21 Updated by SLindoMansilla about 3 years ago

  • Due date changed from 2018-09-25 to 2018-10-09

Moving to sprint 27 since we were not able to resolve it during sprint 26

#22 Updated by SLindoMansilla about 3 years ago

  • Due date changed from 2018-10-09 to 2018-10-23

U-Team agreed that this ticket is usually bigger and more complex than estimated. Let's clarify better and re estimate when the "whole" team is available, on sprint 27 we have 3 people available the whole sprint and 2 for half sprint.

Postponing it to sprint 28

#23 Updated by okurz about 3 years ago

  • Target version changed from Milestone 19 to Milestone 20

#25 Updated by okurz about 3 years ago

Hm, I have an idea: Would it work to just override the unused assets per machine, e.g. on ipmi and s390x-z/VM set ISO= and +ISO_1= (and endless more)?

#26 Updated by riafarov about 3 years ago

+1, oorlov have tested this approach and it works perfectly.

#27 Updated by okurz about 3 years ago

riafarov wrote:

+1, oorlov have tested this approach and it works perfectly.

Just to be sure, how do you define "works perfectly"? I have the following expectations on a working way:

  • clone-job on these jobs would not download useless assets, e.g. no ISO in s390x jobs
  • the assets for other variants are still properly tracked by openQA and belong to other jobs, e.g. the x86_64 non-ipmi ones still track the ISO but the ipmi one not, and therefore also deleted accordingly

#28 Updated by okurz almost 3 years ago

  • Due date deleted (2018-10-23)
  • Target version changed from Milestone 20 to Milestone 21

#29 Updated by okurz almost 3 years ago

According to #44690#note-21 SLindoMansilla removed some "ISO=" setting again from testsuites so maybe we need to do that again after https://github.com/os-autoinst/openQA/pull/1912 applied.

#30 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 21 to Milestone 25+

I would like to see this sooner but right now I do not see the capacity for QSF-u anytime soon

#31 Updated by okurz over 2 years ago

  • Parent task deleted (#38798)

removing #38798 as parent and re-adding as related as the task of the parent was to make sure all assets are accounted for which I think we have. This one is an improvement on top.

#32 Updated by mgriessmeier over 2 years ago

  • Target version changed from Milestone 25+ to Milestone 26

time to check that one again after FCS of 15SP1

#33 Updated by mgriessmeier about 2 years ago

  • Target version changed from Milestone 26 to Milestone 27

to be discussed

#34 Updated by mgriessmeier about 2 years ago

  • Status changed from Workable to New
  • Priority changed from Normal to High
  • Target version changed from Milestone 27 to Milestone 28

for grooming

#35 Updated by dheidler almost 2 years ago

  • Subject changed from [functional][u][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to [medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Status changed from New to Rejected

We don't really understand the ticket.
Please explain it better if you want to reopen.

#36 Updated by okurz almost 2 years ago

  • Subject changed from [medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to [functional][u][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Description updated (diff)
  • Status changed from Rejected to Workable

Updated the description, especially with suggestions. I hope this makes it more clear.

#37 Updated by okurz almost 2 years ago

  • Related to action #59394: [qe-core][functional] Overwrite empty ISO variable everywhere where not needed, i.e. `+ISO=`, to prevent useless ISO downloading and storage added

#38 Updated by mgriessmeier almost 2 years ago

  • Target version changed from Milestone 28 to Milestone 30

needs to be discussed offline

#39 Updated by tjyrinki_suse 12 months ago

  • Subject changed from [functional][u][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to [qe-core][functional][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used

#40 Updated by szarate 8 months ago

  • Project changed from openQA Tests to openQA Project
  • Subject changed from [qe-core][functional][medium] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Category deleted (Enhancement to existing tests)

It rather belongs to the tools team (If it's still valid)

#41 Updated by szarate 8 months ago

  • Target version deleted (Milestone 30)

#42 Updated by okurz 8 months ago

  • Category set to Support
  • Status changed from Workable to Feedback
  • Assignee set to okurz
  • Target version set to Ready

szarate wrote:

It rather belongs to the tools team (If it's still valid)

szarate: Well, I don't think so but maybe you can explain better how you think we could help? I tried with the suggestions in the ticket to make it more clear what would need to be done from the side of test settings and regarding os-autoinst-distri-opensuse, I don't see where this would need anything more from tooling side but I am happy to learn about your opinion then.

#43 Updated by szarate 8 months ago

okurz wrote:

szarate wrote:

It rather belongs to the tools team (If it's still valid)

szarate: Well, I don't think so but maybe you can explain better how you think we could help? I tried with the suggestions in the ticket to make it more clear what would need to be done from the side of test settings and regarding os-autoinst-distri-opensuse, I don't see where this would need anything more from tooling side but I am happy to learn about your opinion then.

One quick idea that came during the meeting, was for instance:

AC3: Jobs without ISO do not have the setting ISO_MAXSIZE

Have the api call to check for test_suites that allows to do such search.

For the others, there were too many unknowns on how to check for accounted/unaccounted assets without doing a lot of manual/busy work (i.e lack of API/knowledge on how to implement such checks programmatically)

#44 Updated by okurz 8 months ago

  • Assignee changed from okurz to szarate

szarate wrote:

okurz wrote:

szarate wrote:

It rather belongs to the tools team (If it's still valid)

szarate: Well, I don't think so but maybe you can explain better how you think we could help? I tried with the suggestions in the ticket to make it more clear what would need to be done from the side of test settings and regarding os-autoinst-distri-opensuse, I don't see where this would need anything more from tooling side but I am happy to learn about your opinion then.

One quick idea that came during the meeting, was for instance:

AC3: Jobs without ISO do not have the setting ISO_MAXSIZE

Have the api call to check for test_suites that allows to do such search.

You can try

openqa-cli api --osd --pretty test_suites | jq '.TestSuites[] | select(.settings[].key == "ISO_MAXSIZE") | {name: .name, id: .id}'

to have a list of all test suites that specify ISO_MAXSIZE. Some are obvious examples where an ISO does not make sense, e.g. "online_sles15sp2_scc_base-srv_all_full_spvm" using PowerVM where an AFAIK ISO can not be directly used. Interesting enough when I call https://openqa.suse.de/tests?match=online_sles15sp2_scc_base-srv_all_full_spvm I don't find any job using that test suite so there are even test suites that are not used at all :(

In general my understanding is that "QE Core" is responsible to ensure a good management of test definitions on OSD, e.g. enforce rules for other teams and such. How about alerts in grafana if there are any unused test suites and references to ISOs when they are not necessary? To remind: The benefit in case this ticket can be resolved would be no useless transfers of unused ISO files back and forth between webUI and workers as well as when cloning to local machines and also less occupied space that could be used for other, more valuable assets, e.g. more qcow files for debugging. If you have good ideas what we can do in "upstream openQA" then this is something which I can of course help more with :)

#45 Updated by okurz 7 months ago

  • Project changed from openQA Project to openQA Tests
  • Subject changed from Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to [qe-core] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Category changed from Support to Enhancement to existing tests
  • Assignee changed from szarate to tjyrinki_suse
  • Target version deleted (Ready)

can you agree with #38807#note-44 ?

#46 Updated by szarate 7 months ago

  • Project changed from openQA Tests to openQA Project
  • Subject changed from [qe-core] Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used to Make sure all used assets are accounted for in tests on osd (#38804) but not as assets which people need to download on cloning the job as the ISO is not actually used
  • Category changed from Enhancement to existing tests to Support
  • Assignee changed from tjyrinki_suse to szarate

okurz wrote:

can you agree with #38807#note-44 ?

no, that's "Solving a problem by tinkering with CLI", how about providing the API call that allows to search for test suites by one or more settings?

If you don't agree, please reject the ticket, instead of playing ping pong.

#47 Updated by okurz 6 months ago

  • Category changed from Support to Feature requests
  • Status changed from Feedback to New
  • Assignee deleted (szarate)
  • Priority changed from High to Low
  • Target version set to future
  • Estimated time deleted (8.00 h)

ok, fine. I can move this ticket back as a potential future task to follow up as SUSE QE Tools. Please keep in mind that I had originally created this ticket as PO for QSF as I see the benefit for test maintainers.

As the settings for openqa.suse.de are SUSE internal we can not easily come up with CI checks in os-autoinst-distri-opensuse that prevent others from adding to the problem. We could come up with CI checks in gitlab though, accept the current state and as a first prevent it from getting worse. What I see as unfortunate though is that many people accept e.g. that assets are needlessly transferred back and forth and extending the time it takes to conduct tests both personally as well as on the production instances. And the SUSE QE Tools team is actually the team that suffers the least from such issues, all other users do though so there should be inherent interest.

#48 Updated by szarate 6 months ago

okurz wrote:

ok, fine. I can move this ticket back as a potential future task to follow up as SUSE QE Tools. Please keep in mind that I had originally created this ticket as PO for QSF as I see the benefit for test maintainers.

I also see the benefit for test mantainers, but if we only have tinkering on the user side, improving the usability/scriptability of openQA will be harder; See my last sentence in this comment.

As the settings for openqa.suse.de are SUSE internal we can not easily come up with CI checks in os-autoinst-distri-opensuse that prevent others from adding to the problem. We could come up with CI checks in gitlab though, accept the current state and as a first prevent it from getting worse.

See my last sentence in this comment.

What I see as unfortunate though is that many people accept e.g. that assets are needlessly transferred back and forth and extending the time it takes to conduct tests both personally as well as on the production instances.

Don't generalize.

And the SUSE QE Tools team is actually the team that suffers the least from such issues, all other users do though so there should be inherent interest.

How about providing the API call that allows to search for test suites by one or more settings? and possibly to filter either with perl expressions/operators or something that allows such behavior

eg: openqa-cli api --json --data '{"ISO_SIZE": "gt 0","HDD_1" :"ne ''", "ISO":"''", "BOOT_FROM_HDD" : "eq 1"}' --osd test-suites perl operators (with a huge security hole but you know the intent)
or: openqa-cli api --json --data '{"ISO_SIZE": ".*","HDD_1" :".*", "ISO":".*", "BOOT_FROM_HDD" : "1"}' --osd test-suites regexp

Once this is done, I think the ticket could be made workable again for test mantainers.

#49 Updated by okurz 6 months ago

Hm, are you sure that a custom query language would be easier to learn for users than using general, common tools that are much better at their specific job?

#50 Updated by szarate 6 months ago

okurz wrote:

Hm, are you sure that a custom query language would be easier to learn for users than using general, common tools that are much better at their specific job?

You don't need to define a custom query language, perl expressions should work fine, or even regular expressions, there's absolutely no need to create a new DSL :), then users can be much happier when performing queries and having to use less tools for such job.

Also available in: Atom PDF