action #48182

[openqa] Disable bug carry over for a job group

Added by rpalethorpe about 1 year ago. Updated about 1 year ago.

Status:ResolvedStart date:21/02/2019
Priority:HighDue date:
Assignee:mkittler% Done:

0%

Category:Feature requests
Target version:Current Sprint
Difficulty:easy
Duration:

Description

Because we have an external script for propagating bug tags, we need to remove OpenQA's carry over comments.

In fact OpenQA's carry over comments are almost always wrong for the Kernel group anyway. They have also begun dropping the message stating they are a carry over comment. This makes deleting them more challenging. For example I did not post the following comment https://openqa.suse.de/tests/2481835#comment-165567 and neither did my script (you can see the bug summary is stale).

Alternatively to disabling the comments, we could parse any existing comments and check if they are doing something suspicious, like tagging a passing test. Then delete/modify those comments, however this could result in legitimate comments being deleted (or modified) in corner cases. The user should be able to override the script so I don't think this is a good idea.

AFAICT it is not currently possible to disable bug carry over for a given job group.

History

#2 Updated by coolo about 1 year ago

  • Project changed from SUSE QA tests to openQA Project
  • Category set to Feature requests
  • Priority changed from Normal to High
  • Target version set to Ready
  • Difficulty set to easy

#3 Updated by okurz about 1 year ago

Alternatively to disabling the comments, we could parse any existing comments and check if they are doing something suspicious, like tagging a passing test

bug carry over is not happening for passed tests. Did you observe something different?

Please keep also the following points in mind when disabling bug carry over for a job group:

  • The black certificate icon will not have a use for that job group because with unlabeled jobs the job group and index page can not show if all failures are attended as documented on http://open.qa/docs/#_review_badges
  • Additional external reporting tools, e.g. http://github.com/os-autoinst/openqa_review, rely on the job labels. When you do not want to use them anymore the accordingly generated reports would give a wrong impression for the corresponding job groups. We could cover that by excluding the corresponding job groups whenever we generate the according reports. openqa-review has a configuration parameter that we already use to exclude e.g. the "Development" and "Staging" job groups
  • https://wiki.suse.net/index.php?title=RD-OPS_QA/openQA_review#Best_practices_for_review should be adapted
  • Other users are used to find the bug references in openQA. If you want to handle it differently I suggest to properly inform upfront. For job groups that are not part of the "build validation" parent groups, e.g. in "Development" that is not a problem though. Easy remedy therefore would be to move the job group, e.g. "Kernel" into "Development" or similar

@rpalethorpe You might want to extract the above points into indvidual tickets

#4 Updated by okurz about 1 year ago

  • Status changed from New to Feedback
  • Assignee set to okurz

I think we wanted to achieve something similar already by redefining the use of labels vs. issue references. I wonder if we missed something there. I would like to crosscheck that.

#5 Updated by okurz about 1 year ago

2d2b18a0 made sure that normal labels are not carried over and as far as I know the messages back-referencing openQA references from other issue trackers which start with label:linked are not carried over automatically.

I would like to understand what you intend to do first:

They have also begun dropping the message stating they are a carry over comment.

Who is "They" in this context?

This makes deleting them more challenging. For example I did not post the following comment https://openqa.suse.de/tests/2481835#comment-165567 and neither did my script (you can see the bug summary is stale).

You are saying that you did not write the comment https://openqa.suse.de/tests/2481835#comment-165567 . Are you saying that openQA wrote it? That would make sense according to
fd450303 which introduced the code

        my $text = $comment->text;
        if ($text !~ "Automatic takeover") {
            $text .= "\n\n(Automatic takeover from t#" . $prev->id . ")\n";
        }

so it would automatically "takeover" or carry-over the comment if it references a bugref but not adding automatic takeover. The purpose was to just not cascade the addition of these footer in subsequent jobs but it seems you try to (ab-)use it for a different purpose?

I guess "deleting all comments in current builds" is not an option for you because you want to still label but by the help of an external script that you run against every job so you do not want openQA to carry over comments, right?

Actually configuring the job group to disable carryover I consider as tricky because a "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA. I would rather go for a special label to disable the carry-over based on the comments. With this it would be way more obvious to readers of the comments why it is not carried-over, exactly how you proposed in #45866#note-9. WDYT?

#6 Updated by rpalethorpe about 1 year ago

okurz wrote:

Alternatively to disabling the comments, we could parse any existing comments and check if they are doing something suspicious, like tagging a passing test


bug carry over is not happening for passed tests. Did you observe something different?

This may be true for pure OpenQA tests and LTP which I dynamically generate test modules for. It is not true of fstests because all the test case results are reported under the same test module. This is just an example though, the problems are never ending and the iteration time in OpenQA is simply too slow for the Kernel&Networking group because we are dealing with a number of external test suites all with different problems.

Please keep also the following points in mind when disabling bug carry over for a job group:

* The black certificate icon will not have a use for that job group because with unlabeled jobs the job group and index page can not show if all failures are attended as documented on http://open.qa/docs/#_review_badges

It is fairly meaningless anyway once you have 6000+ tests and some noise which it may simply be best to ignore rather than tag with some placeholder. On the other hand we are far more likely to achieve the black certificate with JDP than with bug carry over.

  • Additional external reporting tools, e.g. http://github.com/os-autoinst/openqa_review, rely on the job labels. When you do not want to use them anymore the accordingly generated reports would give a wrong impression for the corresponding job groups. We could cover that by excluding the corresponding job groups whenever we generate the according reports. openqa-review has a configuration parameter that we already use to exclude e.g. the "Development" and "Staging" job groups

This script has never worked for the Kernel group. I tried adapting it, but there are some fundamental problems, see https://richiejp.github.io/jdp/development/#Existing-solutions-1. We should discuss recombining efforts at some point, but we actually need separate solutions for different jobs groups and test suites. The only thing we really should share is the data caching/normalisation backend and some other libraries.

You previously said this to me and I changed it to point to PES: https://wiki.suse.net/index.php?title=RD-OPS_QA/openQA_review#Kernel

  • Other users are used to find the bug references in openQA. If you want to handle it differently I suggest to properly inform upfront. For job groups that are not part of the "build validation" parent groups, e.g. in "Development" that is not a problem though.

The only difference is that we include the external test suite name in the bug tag, see https://pes.suse.de/Kernel_Network_QA_Team/openqa_review/ and https://richiejp.github.io/jdp/bugrefs/. I have been broadcasting our problems and the potential solutions on a regular basis.

Easy remedy therefore would be to move the job group, e.g. "Kernel" into "Development" or similar

I really don't care what job group we put our tests in or even where we run them, but this is not acceptable for PMs, atleast for the time being.

How about we tag the job groups with what automation we wish to use and then the review script and JDP can read these to decide where they run?

#7 Updated by rpalethorpe about 1 year ago

I would like to understand what you intend to do first:

They have also begun dropping the message stating they are a carry over comment.


Who is "They" in this context?

OpenQA

This makes deleting them more challenging. For example I did not post the following comment https://openqa.suse.de/tests/2481835#comment-165567 and neither did my script (you can see the bug summary is stale).


You are saying that you did not write the comment https://openqa.suse.de/tests/2481835#comment-165567 . Are you saying that openQA wrote it? That would make sense according to
fd450303 which introduced the code


my $text = $comment->text;
if ($text !~ "Automatic takeover") {
$text .= "\n\n(Automatic takeover from t#" . $prev->id . ")\n";
}

so it would automatically "takeover" or carry-over the comment if it references a bugref but not adding automatic takeover. The purpose was to just not cascade the addition of these footer in subsequent jobs but it seems you try to (ab-)use it for a different purpose?


I guess "deleting all comments in current builds" is not an option for you because you want to still label but by the help of an external script that you run against every job so you do not want openQA to carry over comments, right?

yes.

Actually configuring the job group to disable carryover I consider as tricky because a "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA. I would rather go for a special label to disable the carry-over based on the comments. With this it would be way more obvious to readers of the comments why it is not carried-over, exactly how you proposed in #45866#note-9. WDYT?

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

#8 Updated by rpalethorpe about 1 year ago

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.

I could use label:linked to improve matters, but it is not ideal.

#9 Updated by okurz about 1 year ago

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.

rpalethorpe wrote:

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.

I think we have that and use that. Can you refer to an example where you think this is not the case? The "label:linked" comments are actually coming from openQA itself. However, that in particular could also be mentioned in the comment: That this comment comes from openQA (and not any external script) :)

I could use label:linked to improve matters, but it is not ideal.

Why do you consider a label not ideal?

#10 Updated by rpalethorpe about 1 year ago

okurz wrote:

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913


Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.

I want to label the job group. So would be nice to put the no carry-over label in the job group description or a pinned comment.

Or it could just be a tick box in the job group settings page.

rpalethorpe wrote:

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.


I think we have that and use that. Can you refer to an example where you think this is not the case? The "label:linked" comments are actually coming from openQA itself. However, that in particular could also be mentioned in the comment: That this comment comes from openQA (and not any external script) :)

So ttm is OpenQA itself? Yeah, it would be helpful if such comments linked to information about where they came from and what the labels mean. If OpenQA understands the label it could turn it into a hyperlink, the same as for bugrefs.

I think OpenQA should display its own/propagated comments differently using HTML and CSS. Then we would avoid the issue where it is using a regex on the comment contents to decide whether to add the carry over text.

I could use label:linked to improve matters, but it is not ideal.


Why do you consider a label not ideal?

Because users would need to add the label manually to their comments to prevent them being carried over. Alternatively my script could delete their comments and re-add them with the label, but then a bug in the script could result in a bunch of comments being lost.

#11 Updated by okurz about 1 year ago

rpalethorpe wrote:

okurz wrote:

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913


Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.


I want to label the job group. So would be nice to put the no carry-over label in the job group description or a pinned comment.


Or it could just be a tick box in the job group settings page.

Yes, both is technically possible but I would prefer to not have that as I described: A "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA.

[…]

So ttm is OpenQA itself? Yeah, it would be helpful if such comments linked to information about where they came from and what the labels mean. If OpenQA understands the label it could turn it into a hyperlink, the same as for bugrefs.

True. I was wrong assuming that ttm – which is not openQA itself – would tell where the comment comes from, e.g. see
https://openqa.suse.de/tests/2489129#comment-166372
Would you mind proposing that change on https://github.com/openSUSE/openSUSE-release-tools ?

I could use label:linked to improve matters, but it is not ideal.

Why do you consider a label not ideal?

Because users would need to add the label manually to their comments to prevent them being carried over.

Yes, but then it would be explicit why a comment is not carried-over.

Alternatively my script could delete their comments and re-add them with the label, but then a bug in the script could result in a bunch of comments being lost.

I would be ok with comments being lost in this case.

#12 Updated by mkittler about 1 year ago

Enabling/disabling carry over should only affect bugref carry over in job comments, right? Labels/tags which are added via job group comments would not be affected at all. I'm just asking because such labels/tags were mentioned in the previous discussion.

Controlling this on job group level like the issue description suggests would be the easiest way to implement it. Controlling this per scenario would be more difficult because a scenario is right now not an entity in our database or configuration.

#13 Updated by okurz about 1 year ago

mkittler wrote:

Enabling/disabling carry over should only affect bugref carry over in job comments, right? Labels/tags which are added via job group comments would not be affected at all. I'm just asking because such labels/tags were mentioned in the previous discussion.

Correct, it's about carry-over for bugrefs in jobs. "Build tags" in job groups do not "carry over".

Controlling this on job group level like the issue description suggests would be the easiest way to implement it. Controlling this per scenario would be more difficult because a scenario is right now not an entity in our database or configuration.

And because therefore scenarios do not "belong" to any job group I would also not link jobs or scenarios to a job group.

https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Schema/Result/Jobs.pm#L1663 is the relevant method. I am not exactly sure if comments with labels are skipped even though they might be detected as bugref as commented in #48182#note-5 .

#14 Updated by rpalethorpe about 1 year ago

My understanding from a discussion with Oliver is that we can add a label to each 'scenario' (which I think may also be referred to as a job-template?) by commenting with a label on any of the builds. In the same way you would add a label to an individual build/job. The difference is the label will also be used for future builds. This is a good solution from my point of view (ignoring any implementation details).

Possibly it would be useful to 'pin' such comments so they appear at the top of the comments for all such builds. That can wait until later though.

#15 Updated by rpalethorpe about 1 year ago

  • Status changed from Feedback to In Progress
  • Assignee deleted (okurz)

Coolo pointed out that label:linked has some other side effects which we want to avoid (such as keeping the assets longer); which appears to be true. Then I also found out that IT DOES NOT PREVENT CARRY OVER ANYWAY (https://openqa.suse.de/tests/2525421#comments). This can be seen in the code and from the link. It also sounds like Oliver suspected this, but was happy to propose it anyway.

I think Oliver has pointlessly complicated this. I almost think he is trying to waste our time or prevent us from breaking away from a model/structure he wants to enforce, but we are not in anyway.

So thinking about it more clearly:

  1. This is a really simple anti-feature: disable bug carry over on job groups.

  2. It has a simple clean solution: provide check box, or label to disable carry over on job group.

  3. IF users are confused by the lack of carry over, again, simple solution: display notice/icon in job group and job comments.

In theory this could be done on a per job/scenario basis which gives more control, but the only use case we have is to disable carry over for entire job groups. I don't see the point of complicating it from the user or implementation perspective. Unless someone really thinks it will be required on a per scenario basis or it is actually just simpler technically.

Thinking about second order effects; doing this will cause less people to use the feature. Probably advanced users will disable it and will implement it externally if at all. This will allow it to be optimised and supported as a "sane default" for users with relatively simple setups. So, I don't see any design issue there.

I will remove Oliver as assignee, because I don't believe he is interested in implementing the (anti)feature as described and is being misleading.

#16 Updated by mkittler about 1 year ago

  • Status changed from In Progress to Feedback
  • Assignee set to mkittler

I don't see the point of complicating it from the user or implementation perspective.

Me neither. This would not be hard to implement on job group level. I would store this flag as a column in the job groups table and make it configurable like the other (mostly cleanup related) job group settings. Displaying a note somewhere in the commenting UI should also be straight forward.

So if we can agree on the "per job group" approach I'd start working on it.

#17 Updated by okurz about 1 year ago

@rpalethorpe Sorry if you think I have mislead. I mentioned to you in person on what I would try or you can do. Also I have had a talk with you in person where I explicitly mentioned that I also do not think that labels prevent a carry-over as is. It just looks like the comments by "SYSTEM" are not detected as bugrefs because they might circumvent the bugref parsing which is done for "external commenters". I suggested to add another explicit label and skip the carry-over explicitly when that label is found. I would have removed myself as assignee the next week so of course I am fine with someone else taking over.

#18 Updated by coolo about 1 year ago

Marius - my agreement to the feature is marked in the first comment. I even set it to High priority, but didn't pick it to the backlog right away

#19 Updated by mkittler about 1 year ago

  • Status changed from Feedback to In Progress

Ok. To retain consistency with default priority and cleanup settings I add this on the parent group level as well.

#20 Updated by mkittler about 1 year ago

  • Target version changed from Ready to Current Sprint

#21 Updated by mkittler about 1 year ago

  • Status changed from In Progress to Resolved

PR has been merged

Also available in: Atom PDF