Project

General

Profile

action #65004

The asset generated by FORCE_PUBLISH works fine but does not show up in "Logs & Assets" and is also not mentioned in logs

Added by leli over 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-03-30
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

http://openqa.nue.suse.com/tests/4052264#settings FORCE_PUBLISH_HDD_1=bsc1167756_2.qcow2

The qcow2 can be found in https://openqa.nue.suse.com/assets/hdd/bsc1167756_2.qcow2, but doesn't show up in "Logs & Assets" and is also not mentioned in logs.

Acceptance criteria

  • AC1: Assets specified with FORCE_PUBLISH and uploaded successfully are visible under "Logs & Assets" of job details

History

#1 Updated by okurz over 1 year ago

  • Project changed from openQA Tests to openQA Project
  • Category changed from Bugs in existing tests to Feature requests

#2 Updated by okurz over 1 year ago

  • Description updated (diff)
  • Status changed from New to Workable
  • Target version set to Ready

#3 Updated by Xiaojing_liu over 1 year ago

I tried to reproduce this issue in my local, but could not reproduce it.
Here is my test: http://10.67.19.103/tests/43#downloads
Do I miss something?

#4 Updated by mkittler over 1 year ago

Do I miss something?

The only difference between your test and the one from the ticket description is that yours didn't fail. That might make a difference.


"Logs & Assets" shows all assets associated with a job except hidden ones (and on our instance only repo assets are hidden). The assets to be associated with a job are deduced from the job settings (asset_type_from_setting function) and the settings variables which are considered are documented here. The FORCE_PUBLISH_HDD variable and the regular PUBLISH_HDD variables are not considered here. So I'm currently wondering how these variables are supposed to work at all. Considering @Xiaojing_liu's tests there must be some handling; otherwise it wouldn't have worked. Since a simple grep for PUBLISH_HDD didn't reveal any interesting places this seems to require more reverse engineering unless somebody else knows how this works.


But regardless of that we should ensure that PUBLISH_HDD and FORCE_PUBLISH_HDD are handled consistently.

#5 Updated by okurz over 1 year ago

I also don't know more. I just grepped for "PUBLISH" – not just "PUBLISH_HDD" – and found only a single reference to lib/OpenQA/Schema/ResultSet/Jobs.pm: my @invalid_keys = grep { $_ =~ /^(PUBLISH_HDD|FORCE_PUBLISH_HDD|STORE_HDD)\S+(\d+)$/ && $settings{$_} =~ /\// } where FORCE is already there.

As the asset did show up for Xiaojing_liu I guess the problem lies not in mentions for FORCE_PUBLISH but rather "still record as asset despite the failure"

#6 Updated by mkittler over 1 year ago

The line that you've found (and the corresponding test) are just for avoiding /-characters in those variables.

but rather "still record as asset despite the failure"

yes

#7 Updated by mkittler over 1 year ago

  • Assignee set to mkittler
  • Target version changed from Ready to Current Sprint

#8 Updated by mkittler over 1 year ago

  • Status changed from Workable to Blocked

Note that the asset has apparently been cleaned up in @Xiaojing_liu's test job. But it was there last time I looked at her instance. Likely the asset has been removed by the cleanup (the job is today 14 days old so if she didn't change the default storage duration this is expected).

This let's me of course wonder whether the same has happened for the job mentioned in the job description. However, in this case there's really no trace of the asset being uploaded in the worker log.


It seems there are 2 ways assets can be associated with a job:

  1. From the job settings in register_assets_from_settings which is using the previously mentioned asset_type_from_setting function.
  2. When the worker has uploaded the asset successfully (see OpenQA::WebAPI::Controller::API::V1::Job::create_artefact).

I suppose 1. has no relevance here because none of the PUBLISH variables are handled here and it nevertheless works in other cases.

Looking at the upload/worker code for 2. it seems that the code is completely independent from job settings. The worker just uploads any file which os-autoinst adds to the assets_public or assets_private directories within the pool directory. Considering that there's no trace of the asset being uploaded in the worker log the asset was likely not even created by os-autoinst. That means the web UI and most likely also the worker are not at fault here. So most likely os-autoinst simply did not create the asset at all. And maybe that's even just because the test didn't reach the point where the asset would be generated (it failed already in update).

That leaves the question why the asset was nevertheless found via https://openqa.nue.suse.com/assets/hdd/bsc1167756_2.qcow2 (the link is dead at this point). An explanation for this is that the URL merely returned the asset created by the previous job (which passed).

The theory that os-autoinst didn't create the asset could be easily checked if autoinst-log.txt would be still there but it isn't.

leli Before you can show me a more recent example I would postpone putting further effort into this ticket (especially considering that @Xiaojing_liu already tested that is generally works).


By the way, I was trying to find a recent example myself on OSD and o3 via select * from job_settings where key like 'FORCE_PUBLISH_%' order by t_created desc but couldn't find a recent one.

#9 Updated by okurz over 1 year ago

  • Status changed from Blocked to Workable

mkittler the feature "FORCE_PUBLISH" is probably only used when someone manually triggers investigation jobs. I suggest you just select any failing test scenario from a production scenario where it makes sense and trigger a clone with the parameter set.

#10 Updated by mkittler over 1 year ago

I've been using select * from job_settings join jobs on jobs.id = job_id where key like 'PUBLISH_%' and result = 'failed' order by jobs.t_created desc limit 10; to find a failed job which would have published an image (if it hadn't failed).

I cloned the job via script/openqa-clone-job … https://openqa.suse.de/tests/4288871 FORCE_PUBLISH_HDD_1=SLES-15-SP1-x86_64-mru-install-minimal-with-addons-Build:15220:susefirewall2-to-firewalld-Server-DVD-Incidents-64bit.qcow2 PUBLISH_HDD_1= to turn PUBLISH_HDD_1 into FORCE_PUBLISH_HDD_1. The problem is that the test locally passed but also didn't execute any modules beyond installation.

Not sure whether it is worth putting more effort into figuring the steps to reproduce this issue. Especially considering that likely os-autoinst didn't produce the image here in the first place and FORCE_PUBLISH has nothing to do with it.

#11 Updated by cdywan about 1 year ago

Is this actually reproducible anymore?

#12 Updated by mkittler about 1 year ago

  • Assignee deleted (mkittler)

I could never reproduce it because reproducing it is simply not very straight forward with the information from the ticket description. Maybe I'm also correct and os-autoinst didn't create the asset in this case. I'm unassigning because I've never received any feedback (not even from the author of the ticket) and I assume it does not seem to be very important anymore.

#13 Updated by cdywan about 1 year ago

  • Status changed from Workable to Feedback

#14 Updated by okurz about 1 year ago

  • Status changed from Feedback to Workable
  • Target version changed from Current Sprint to Ready

based on #65004#note-9 and more recent comments I suggest we do not try anymore to reproduce the problem but rather try to show a sucessful use of "FORCE_PUBLISH" in a useful way to verify everything works as expected. E.g. find/create a test scenario that should generate and publish an HDD image but fails and force that scenario to publish an image nevertheless with FORCE_PUBLISH.

#15 Updated by Xiaojing_liu about 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to Xiaojing_liu

#17 Updated by Xiaojing_liu about 1 year ago

  • Status changed from In Progress to Feedback

The pr has been merged

#18 Updated by Xiaojing_liu 12 months ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF