Project

General

Profile

action #14926

(missing docu) job data is cleaned up way too eagerly, e.g. for still open bugs

Added by okurz about 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2016-11-21
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

motivation

Take for example the bug bsc#990254 referencing an openQA job from July. The corresponding job run is https://openqa.suse.de/tests/482779 which is not available anymore. Unfortunately the job information page does not even tell if the link is invalid or has been valid but the job does not exist anymore. Even if the corresponding logfiles would be cleaned already the job would provide at least some information, at least it would show a link to "latest"

user story

As a user referencing job results externally (e.g. in bug reports) I want the linked jobs to stay around for way longer to not loose at least the job details

acceptance criteria

  • AC1: A job used in bug reports is not cleaned as soon as unreferenced jobs. It's considered important
  • AC2: Jobs just marked with a issue link are not considered "important"
  • AC3: Jobs marked automatically by carry-over are not considered "important" (as there was no human action on this job and therefore less valuable)
  • AC4: Deleted job pages show something more reasonable than a page which is just telling that the page does not exist

tasks

  • Define a label to be used or a database entry, e.g. label:linked
  • On cleanup consider the jobs with label:linked as important
  • optional: Automatically add label based on if the page has been called or the buglink has been added there based on human intervention
  • optional: redirect from job not found page to something more helpful, e.g. when the job page probably existed at a time but does not anymore, e.g. because the job number is smaller than the current existing jobs

original suggestion

  1. Do not delete jobs which have a bugref 2a. Do not completely delete old jobs or 2b. show a custom details page for old jobs with at least some explanation of what is happening, e.g. link to latest

Related issues

Related to openQA Project - action #12266: keep tests with any comment around laterRejected2016-06-08

History

#1 Updated by coolo about 6 years ago

  • 1. is unfeasible - too many jobs have a bug reference thanks to carry over. And unfortunately having a bug reference or not does not relate to the test linked somewhere
  • 2a. you will need to specify what not to remove
  • 2b. link to what latest? The only information left from a removed job is that it's not there

#2 Updated by okurz about 6 years ago

  • Description updated (diff)
  • Category changed from Concrete Bugs to Feature requests

We discussed in a call some ideas:

  • use labels, e.g. label:linked for every job that was linked in a bug report and keep these jobs around longer as "important" ones, same as all jobs that belong to "important" builds.
  • use automatic script based on apache logs or maybe from within openQA when a job page is called and it has a bug reference which has not been added by automatic means, consider this job as "important" by putting the automatic label in the comment.

#3 Updated by oholecek about 6 years ago

  • Assignee set to oholecek

I'll into this. I'm for tracking referral header instead of apache log parsing. How about adding label:linked for all jobs with referral header present? I still need to look into how apache proxy handles them though.

#4 Updated by oholecek about 6 years ago

I have one issue with this though and that updating job label if referer is present violates REST rules. Get request should not modify data.

What are your opinions on this?

#5 Updated by coolo about 6 years ago

As long as you don't use the credentionals of the one GETing, I see no difference to the approach with parsing logs

#6 Updated by oholecek about 6 years ago

As I am updating comment user queries, I'm wondering do we have something against creating system user?

Currently I'm taking same approach as in case of audit log where event owner db field is nullable and there are explicit checks for valid user. Comments does not have nullable user yet -> schema migration. And there are at least 5 occurrences of $user->... which needs explicit user checking.

Whereas if I check and create system user on webui startup I can use its id for comments instead of undef. Thus no need for migration and unnecessary code changes. This system user wont have proper openid URL thus there is no conflict with existing users. Also I can use fixed string for gravatar -> all openqa instances should have the same picture for admin user from gravatar.

I'm inclined to add this user. WDYT?

#7 Updated by oholecek about 6 years ago

  • Status changed from New to In Progress

PR with job labeling and expiration merged PR#1060. What is missing is the flagging of label carry-over and considering all labeled jobs important

#8 Updated by okurz about 6 years ago

And some documentation should be added, at least on https://progress.opensuse.org/projects/openqav3/wiki if not directly within the documentation of openQA

#9 Updated by okurz about 6 years ago

https://github.com/os-autoinst/openQA/pull/1071 changes the behaviour for labels. They are not carried over anymore. Please think hard if this is what we want.

#10 Updated by okurz about 6 years ago

https://github.com/os-autoinst/openQA/pull/1071 merged, what do we need to do now?

#11 Updated by coolo about 6 years ago

updating the documentation I'd say

#12 Updated by okurz about 6 years ago

we agreed in the call that szarate will provide his enhancements to the documentation until start of next week. Next step is then to add some admin document also explaining what is happening here.

#13 Updated by szarate about 6 years ago

In the meantime, so this doesn't gets blocked, it can be a separate document. Until a proper adminguide is written (Or simply add this as part of it, as the first topic, then we could move stuff like Installing.asciidoc#other-database-engines there (If it makes sense also)

#14 Updated by okurz about 6 years ago

szarate don't try to get out of there. I deliberately mentioned that we are blocked by you to give some additional motivation ;-P

#15 Updated by szarate about 6 years ago

okurz good one!, Will give some love to that then :)

#16 Updated by coolo about 6 years ago

  • Subject changed from job data is cleaned up way too eagerly, e.g. for still open bugs to (missing docu) job data is cleaned up way too eagerly, e.g. for still open bugs
  • Priority changed from Urgent to Normal

As the feature itself is done and merged, I move down the priority

#17 Updated by okurz about 6 years ago

  • Target version set to Milestone 5

#18 Updated by okurz about 6 years ago

  • Related to action #12266: keep tests with any comment around later added

#19 Updated by oholecek about 6 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF