action #65004
closedThe asset generated by FORCE_PUBLISH works fine but does not show up in "Logs & Assets" and is also not mentioned in logs
0%
Description
Observation¶
http://openqa.nue.suse.com/tests/4052264#settings FORCE_PUBLISH_HDD_1=bsc1167756_2.qcow2
The qcow2 can be found in https://openqa.nue.suse.com/assets/hdd/bsc1167756_2.qcow2, but doesn't show up in "Logs & Assets" and is also not mentioned in logs.
Acceptance criteria¶
- AC1: Assets specified with FORCE_PUBLISH and uploaded successfully are visible under "Logs & Assets" of job details
Updated by okurz over 4 years ago
- Project changed from openQA Tests (public) to openQA Project (public)
- Category changed from Bugs in existing tests to Feature requests
Updated by okurz over 4 years ago
- Description updated (diff)
- Status changed from New to Workable
- Target version set to Ready
Updated by Xiaojing_liu over 4 years ago
I tried to reproduce this issue in my local, but could not reproduce it.
Here is my test: http://10.67.19.103/tests/43#downloads
Do I miss something?
Updated by mkittler over 4 years ago
Do I miss something?
The only difference between your test and the one from the ticket description is that yours didn't fail. That might make a difference.
"Logs & Assets" shows all assets associated with a job except hidden ones (and on our instance only repo assets are hidden). The assets to be associated with a job are deduced from the job settings (asset_type_from_setting
function) and the settings variables which are considered are documented here. The FORCE_PUBLISH_HDD
variable and the regular PUBLISH_HDD
variables are not considered here. So I'm currently wondering how these variables are supposed to work at all. Considering @Xiaojing_liu's tests there must be some handling; otherwise it wouldn't have worked. Since a simple grep for PUBLISH_HDD
didn't reveal any interesting places this seems to require more reverse engineering unless somebody else knows how this works.
But regardless of that we should ensure that PUBLISH_HDD
and FORCE_PUBLISH_HDD
are handled consistently.
Updated by okurz over 4 years ago
I also don't know more. I just grepped for "PUBLISH" – not just "PUBLISH_HDD" – and found only a single reference to lib/OpenQA/Schema/ResultSet/Jobs.pm: my @invalid_keys = grep { $_ =~ /^(PUBLISH_HDD|FORCE_PUBLISH_HDD|STORE_HDD)\S+(\d+)$/ && $settings{$_} =~ /\// }
where FORCE is already there.
As the asset did show up for Xiaojing_liu I guess the problem lies not in mentions for FORCE_PUBLISH but rather "still record as asset despite the failure"
Updated by mkittler over 4 years ago
The line that you've found (and the corresponding test) are just for avoiding /
-characters in those variables.
but rather "still record as asset despite the failure"
yes
Updated by mkittler over 4 years ago
- Assignee set to mkittler
- Target version changed from Ready to Current Sprint
Updated by mkittler over 4 years ago
- Status changed from Workable to Blocked
Note that the asset has apparently been cleaned up in @Xiaojing_liu's test job. But it was there last time I looked at her instance. Likely the asset has been removed by the cleanup (the job is today 14 days old so if she didn't change the default storage duration this is expected).
This let's me of course wonder whether the same has happened for the job mentioned in the job description. However, in this case there's really no trace of the asset being uploaded in the worker log.
It seems there are 2 ways assets can be associated with a job:
- From the job settings in
register_assets_from_settings
which is using the previously mentionedasset_type_from_setting
function. - When the worker has uploaded the asset successfully (see
OpenQA::WebAPI::Controller::API::V1::Job::create_artefact
).
I suppose 1. has no relevance here because none of the PUBLISH
variables are handled here and it nevertheless works in other cases.
Looking at the upload/worker code for 2. it seems that the code is completely independent from job settings. The worker just uploads any file which os-autoinst adds to the assets_public
or assets_private
directories within the pool directory. Considering that there's no trace of the asset being uploaded in the worker log the asset was likely not even created by os-autoinst. That means the web UI and most likely also the worker are not at fault here. So most likely os-autoinst simply did not create the asset at all. And maybe that's even just because the test didn't reach the point where the asset would be generated (it failed already in update
).
That leaves the question why the asset was nevertheless found via https://openqa.nue.suse.com/assets/hdd/bsc1167756_2.qcow2 (the link is dead at this point). An explanation for this is that the URL merely returned the asset created by the previous job (which passed).
The theory that os-autoinst didn't create the asset could be easily checked if autoinst-log.txt would be still there but it isn't.
@leli Before you can show me a more recent example I would postpone putting further effort into this ticket (especially considering that @Xiaojing_liu already tested that is generally works).
By the way, I was trying to find a recent example myself on OSD and o3 via select * from job_settings where key like 'FORCE_PUBLISH_%' order by t_created desc
but couldn't find a recent one.
Updated by okurz over 4 years ago
- Status changed from Blocked to Workable
@mkittler the feature "FORCE_PUBLISH" is probably only used when someone manually triggers investigation jobs. I suggest you just select any failing test scenario from a production scenario where it makes sense and trigger a clone with the parameter set.
Updated by mkittler over 4 years ago
I've been using select * from job_settings join jobs on jobs.id = job_id where key like 'PUBLISH_%' and result = 'failed' order by jobs.t_created desc limit 10;
to find a failed job which would have published an image (if it hadn't failed).
I cloned the job via script/openqa-clone-job … https://openqa.suse.de/tests/4288871 FORCE_PUBLISH_HDD_1=SLES-15-SP1-x86_64-mru-install-minimal-with-addons-Build:15220:susefirewall2-to-firewalld-Server-DVD-Incidents-64bit.qcow2 PUBLISH_HDD_1=
to turn PUBLISH_HDD_1
into FORCE_PUBLISH_HDD_1
. The problem is that the test locally passed but also didn't execute any modules beyond installation
.
Not sure whether it is worth putting more effort into figuring the steps to reproduce this issue. Especially considering that likely os-autoinst didn't produce the image here in the first place and FORCE_PUBLISH
has nothing to do with it.
Updated by mkittler over 4 years ago
- Assignee deleted (
mkittler)
I could never reproduce it because reproducing it is simply not very straight forward with the information from the ticket description. Maybe I'm also correct and os-autoinst didn't create the asset in this case. I'm unassigning because I've never received any feedback (not even from the author of the ticket) and I assume it does not seem to be very important anymore.
Updated by okurz over 4 years ago
- Status changed from Feedback to Workable
- Target version changed from Current Sprint to Ready
based on #65004#note-9 and more recent comments I suggest we do not try anymore to reproduce the problem but rather try to show a sucessful use of "FORCE_PUBLISH" in a useful way to verify everything works as expected. E.g. find/create a test scenario that should generate and publish an HDD image but fails and force that scenario to publish an image nevertheless with FORCE_PUBLISH.
Updated by Xiaojing_liu about 4 years ago
- Status changed from Workable to In Progress
- Assignee set to Xiaojing_liu
Updated by Xiaojing_liu about 4 years ago
Updated by Xiaojing_liu about 4 years ago
- Status changed from In Progress to Feedback
The pr has been merged
Updated by Xiaojing_liu about 4 years ago
- Status changed from Feedback to Resolved
verify run on OSD:https://openqa.suse.de/tests/4850769/file/autoinst-log.txt