QA » openQA Project

Alternatively to disabling the comments, we could parse any existing comments and check if they are doing something suspicious, like tagging a passing test

bug carry over is not happening for passed tests. Did you observe something different?

Please keep also the following points in mind when disabling bug carry over for a job group:

• The black certificate icon will not have a use for that job group because with unlabeled jobs the job group and index page can not show if all failures are attended as documented on http://open.qa/docs/#_review_badges
• Additional external reporting tools, e.g. http://github.com/os-autoinst/openqa_review, rely on the job labels. When you do not want to use them anymore the accordingly generated reports would give a wrong impression for the corresponding job groups. We could cover that by excluding the corresponding job groups whenever we generate the according reports. openqa-review has a configuration parameter that we already use to exclude e.g. the "Development" and "Staging" job groups
• https://wiki.suse.net/index.php?title=RD-OPS_QA/openQA_review#Best_practices_for_review should be adapted
• Other users are used to find the bug references in openQA. If you want to handle it differently I suggest to properly inform upfront. For job groups that are not part of the "build validation" parent groups, e.g. in "Development" that is not a problem though. Easy remedy therefore would be to move the job group, e.g. "Kernel" into "Development" or similar

@rpalethorpe You might want to extract the above points into indvidual tickets

2d2b18a0 made sure that normal labels are not carried over and as far as I know the messages back-referencing openQA references from other issue trackers which start with label:linked are not carried over automatically.

I would like to understand what you intend to do first:

They have also begun dropping the message stating they are a carry over comment.

Who is "They" in this context?

This makes deleting them more challenging. For example I did not post the following comment https://openqa.suse.de/tests/2481835#comment-165567 and neither did my script (you can see the bug summary is stale).

You are saying that you did not write the comment https://openqa.suse.de/tests/2481835#comment-165567 . Are you saying that openQA wrote it? That would make sense according to
[fd450303]((https://github.com/os-autoinst/openQA/commit/fd450303) which introduced the code

        my $text = $comment->text;
        if ($text !~ "Automatic takeover") {
            $text .= "\n\n(Automatic takeover from t#" . $prev->id . ")\n";
        }

so it would automatically "takeover" or carry-over the comment if it references a bugref but not adding automatic takeover. The purpose was to just not cascade the addition of these footer in subsequent jobs but it seems you try to (ab-)use it for a different purpose?

I guess "deleting all comments in current builds" is not an option for you because you want to still label but by the help of an external script that you run against every job so you do not want openQA to carry over comments, right?

Actually configuring the job group to disable carryover I consider as tricky because a "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA. I would rather go for a special label to disable the carry-over based on the comments. With this it would be way more obvious to readers of the comments why it is not carried-over, exactly how you proposed in #45866#note-9. WDYT?

okurz wrote:

Alternatively to disabling the comments, we could parse any existing comments and check if they are doing something suspicious, like tagging a passing test

bug carry over is not happening for passed tests. Did you observe something different?

This may be true for pure OpenQA tests and LTP which I dynamically generate test modules for. It is not true of fstests because all the test case results are reported under the same test module. This is just an example though, the problems are never ending and the iteration time in OpenQA is simply too slow for the Kernel&Networking group because we are dealing with a number of external test suites all with different problems.

Please keep also the following points in mind when disabling bug carry over for a job group:

• The black certificate icon will not have a use for that job group because with unlabeled jobs the job group and index page can not show if all failures are attended as documented on http://open.qa/docs/#_review_badges

It is fairly meaningless anyway once you have 6000+ tests and some noise which it may simply be best to ignore rather than tag with some placeholder. On the other hand we are far more likely to achieve the black certificate with JDP than with bug carry over.

• Additional external reporting tools, e.g. http://github.com/os-autoinst/openqa_review, rely on the job labels. When you do not want to use them anymore the accordingly generated reports would give a wrong impression for the corresponding job groups. We could cover that by excluding the corresponding job groups whenever we generate the according reports. openqa-review has a configuration parameter that we already use to exclude e.g. the "Development" and "Staging" job groups

This script has never worked for the Kernel group. I tried adapting it, but there are some fundamental problems, see https://richiejp.github.io/jdp/development/#Existing-solutions-1. We should discuss recombining efforts at some point, but we actually need separate solutions for different jobs groups and test suites. The only thing we really should share is the data caching/normalisation backend and some other libraries.

• https://wiki.suse.net/index.php?title=RD-OPS_QA/openQA_review#Best_practices_for_review should be adapted

You previously said this to me and I changed it to point to PES: https://wiki.suse.net/index.php?title=RD-OPS_QA/openQA_review#Kernel

• Other users are used to find the bug references in openQA. If you want to handle it differently I suggest to properly inform upfront. For job groups that are not part of the "build validation" parent groups, e.g. in "Development" that is not a problem though.

The only difference is that we include the external test suite name in the bug tag, see https://pes.suse.de/Kernel_Network_QA_Team/openqa_review/ and https://richiejp.github.io/jdp/bugrefs/. I have been broadcasting our problems and the potential solutions on a regular basis.

Easy remedy therefore would be to move the job group, e.g. "Kernel" into "Development" or similar

I really don't care what job group we put our tests in or even where we run them, but this is not acceptable for PMs, atleast for the time being.

How about we tag the job groups with what automation we wish to use and then the review script and JDP can read these to decide where they run?

I would like to understand what you intend to do first:

They have also begun dropping the message stating they are a carry over comment.

Who is "They" in this context?

OpenQA

This makes deleting them more challenging. For example I did not post the following comment https://openqa.suse.de/tests/2481835#comment-165567 and neither did my script (you can see the bug summary is stale).

You are saying that you did not write the comment https://openqa.suse.de/tests/2481835#comment-165567 . Are you saying that openQA wrote it? That would make sense according to
[fd450303]((https://github.com/os-autoinst/openQA/commit/fd450303) which introduced the code

        my $text = $comment->text;
        if ($text !~ "Automatic takeover") {
            $text .= "\n\n(Automatic takeover from t#" . $prev->id . ")\n";
        }

so it would automatically "takeover" or carry-over the comment if it references a bugref but not adding automatic takeover. The purpose was to just not cascade the addition of these footer in subsequent jobs but it seems you try to (ab-)use it for a different purpose?

I guess "deleting all comments in current builds" is not an option for you because you want to still label but by the help of an external script that you run against every job so you do not want openQA to carry over comments, right?

yes.

Actually configuring the job group to disable carryover I consider as tricky because a "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA. I would rather go for a special label to disable the carry-over based on the comments. With this it would be way more obvious to readers of the comments why it is not carried-over, exactly how you proposed in #45866#note-9. WDYT?

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.

I could use label:linked to improve matters, but it is not ideal.

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.

rpalethorpe wrote:

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.

I think we have that and use that. Can you refer to an example where you think this is not the case? The "label:linked" comments are actually coming from openQA itself. However, that in particular could also be mentioned in the comment: That this comment comes from openQA (and not any external script) :)

I could use label:linked to improve matters, but it is not ideal.

Why do you consider a label not ideal?

okurz wrote:

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.

I want to label the job group. So would be nice to put the no carry-over label in the job group description or a pinned comment.

Or it could just be a tick box in the job group settings page.

rpalethorpe wrote:

Also it would be useful if external scripts linked to themselves when they post comments or detect a tag which activates them.

I think we have that and use that. Can you refer to an example where you think this is not the case? The "label:linked" comments are actually coming from openQA itself. However, that in particular could also be mentioned in the comment: That this comment comes from openQA (and not any external script) :)

So ttm is OpenQA itself? Yeah, it would be helpful if such comments linked to information about where they came from and what the labels mean. If OpenQA understands the label it could turn it into a hyperlink, the same as for bugrefs.

I think OpenQA should display its own/propagated comments differently using HTML and CSS. Then we would avoid the issue where it is using a regex on the comment contents to decide whether to add the carry over text.

I could use label:linked to improve matters, but it is not ideal.

Why do you consider a label not ideal?

Because users would need to add the label manually to their comments to prevent them being carried over. Alternatively my script could delete their comments and re-add them with the label, but then a bug in the script could result in a bunch of comments being lost.

rpalethorpe wrote:

okurz wrote:

rpalethorpe wrote:

If it is a label similar to this: https://openqa.suse.de/group_overview/110#comment-112913

Do you mean tag:144.3:publishing:publishing? That is a build tag. A bit different.

I want to label the job group. So would be nice to put the no carry-over label in the job group description or a pinned comment.

Or it could just be a tick box in the job group settings page.

Yes, both is technically possible but I would prefer to not have that as I described: A "scenario" is independant of the job group and a carryover makes sense per "scenario" and has no relation to the job group. Adding this and making it obvious to users of openQA why a carry over happens or does not happen would be not obvious and probably not fit well into the overall design of openQA.

[…]
So ttm is OpenQA itself? Yeah, it would be helpful if such comments linked to information about where they came from and what the labels mean. If OpenQA understands the label it could turn it into a hyperlink, the same as for bugrefs.

True. I was wrong assuming that ttm – which is not openQA itself – would tell where the comment comes from, e.g. see
https://openqa.suse.de/tests/2489129#comment-166372
Would you mind proposing that change on https://github.com/openSUSE/openSUSE-release-tools ?

I could use label:linked to improve matters, but it is not ideal.
Why do you consider a label not ideal?
Because users would need to add the label manually to their comments to prevent them being carried over.

Yes, but then it would be explicit why a comment is not carried-over.

Alternatively my script could delete their comments and re-add them with the label, but then a bug in the script could result in a bunch of comments being lost.

I would be ok with comments being lost in this case.

Enabling/disabling carry over should only affect bugref carry over in job comments, right? Labels/tags which are added via job group comments would not be affected at all. I'm just asking because such labels/tags were mentioned in the previous discussion.

Controlling this on job group level like the issue description suggests would be the easiest way to implement it. Controlling this per scenario would be more difficult because a scenario is right now not an entity in our database or configuration.

mkittler wrote:

Enabling/disabling carry over should only affect bugref carry over in job comments, right? Labels/tags which are added via job group comments would not be affected at all. I'm just asking because such labels/tags were mentioned in the previous discussion.

Correct, it's about carry-over for bugrefs in jobs. "Build tags" in job groups do not "carry over".

Controlling this on job group level like the issue description suggests would be the easiest way to implement it. Controlling this per scenario would be more difficult because a scenario is right now not an entity in our database or configuration.

And because therefore scenarios do not "belong" to any job group I would also not link jobs or scenarios to a job group.

https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Schema/Result/Jobs.pm#L1663 is the relevant method. I am not exactly sure if comments with labels are skipped even though they might be detected as bugref as commented in #48182#note-5 .

My understanding from a discussion with Oliver is that we can add a label to each 'scenario' (which I think may also be referred to as a job-template?) by commenting with a label on any of the builds. In the same way you would add a label to an individual build/job. The difference is the label will also be used for future builds. This is a good solution from my point of view (ignoring any implementation details).

Possibly it would be useful to 'pin' such comments so they appear at the top of the comments for all such builds. That can wait until later though.

@rpalethorpe Sorry if you think I have mislead. I mentioned to you in person on what I would try or you can do. Also I have had a talk with you in person where I explicitly mentioned that I also do not think that labels prevent a carry-over as is. It just looks like the comments by "SYSTEM" are not detected as bugrefs because they might circumvent the bugref parsing which is done for "external commenters". I suggested to add another explicit label and skip the carry-over explicitly when that label is found. I would have removed myself as assignee the next week so of course I am fine with someone else taking over.

Marius - my agreement to the feature is marked in the first comment. I even set it to High priority, but didn't pick it to the backlog right away