Project

General

Profile

action #45746

Needle selection shows 100% for matched needles (even if 98%)

Added by zgao over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2019-01-07
Due date:
% Done:

0%

Estimated time:
Difficulty:
easy
Duration:

Description

In this case on o3, the needle seems to be matched at similarity 98%, but the dropdown shows a similarity of 100% ---- which is not correct.

I thought of the similarity threshold of needles but in this needle, gnome-screenlock-password-Tumbleweed-20181012, no similarity threshold has been specified. So what is the default threshold for needle matching?

I presume it's a openQA bug, what do you think?

History

#1 Updated by zgao over 1 year ago

As zcjia kindly points out, default similarity threshold seems to be 96% as source code shows.

Can we say it is buggy when needles shows similarity 98% while dropdown shows similarity 100%?

#2 Updated by coolo over 1 year ago

  • Subject changed from Needle falsely matched 100% on o.o.o to Needle selection shows 100% for matched needles (even if 98%)
  • Category set to Concrete Bugs
  • Target version set to Ready
  • Difficulty set to easy

It looks like we don't store an "error" into the json for matched needles. But this is used to calculcate the percentage. So no error means 0, means 100%.

#3 Updated by mkittler over 1 year ago

  • Assignee set to mkittler

#4 Updated by okurz over 1 year ago

And how do you plan to solve this?

I read the value in the pull down menu as: "Matches 100% relative to the match level or higher". So if you have a match level of 96% as per default then there can be an actual, absolute match of 96% up to 100% all considered as "passed"

#5 Updated by coolo over 1 year ago

How you read it is irrelevant to the truth :)

The matchlevel is per area, so there is no way to express multi-area match levels 'relative to the match level' - not in a sane way at least. But let's consider the normal case when every area has the default, 95% should be displayed as 95% and 97% as 97%, not a suprise jump. If the match rate was relative to the match level, then it should also be in the needle area display.

But that again would make it impossible to calculate a good match level. So KISS - and display 98% for a needle with one area matching 98%.

#6 Updated by mkittler over 1 year ago

  • Status changed from New to In Progress
  • Target version changed from Ready to Current Sprint

I agree with @coolo. That's how I'm currently implementing it.

It looks like we don't store an "error" into the json for matched needles.

I've just imported that step result from o3 locally and added the "error" info which is indeed missing. That instantly let's the web UI show the percentage. So it is a bug/limitation only on the os-autoinst side.

Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?

#7 Updated by okurz over 1 year ago

mkittler wrote:

Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?

I would go with "os-autoinst"

#9 Updated by mkittler over 1 year ago

  • Status changed from In Progress to Resolved

PR merged

Also available in: Atom PDF