action #45746
closedNeedle selection shows 100% for matched needles (even if 98%)
0%
Description
In this case on o3, the needle seems to be matched at similarity 98%, but the dropdown shows a similarity of 100% ---- which is not correct.
I thought of the similarity threshold of needles but in this needle, gnome-screenlock-password-Tumbleweed-20181012, no similarity threshold has been specified. So what is the default threshold for needle matching?
I presume it's a openQA bug, what do you think?
Updated by zgao over 5 years ago
As zcjia kindly points out, default similarity threshold seems to be 96% as source code shows.
Can we say it is buggy when needles shows similarity 98% while dropdown shows similarity 100%?
Updated by coolo over 5 years ago
- Subject changed from Needle falsely matched 100% on o.o.o to Needle selection shows 100% for matched needles (even if 98%)
- Category set to Regressions/Crashes
- Target version set to Ready
- Difficulty set to easy
It looks like we don't store an "error" into the json for matched needles. But this is used to calculcate the percentage. So no error means 0, means 100%.
Updated by okurz over 5 years ago
And how do you plan to solve this?
I read the value in the pull down menu as: "Matches 100% relative to the match level or higher". So if you have a match level of 96% as per default then there can be an actual, absolute match of 96% up to 100% all considered as "passed"
Updated by coolo over 5 years ago
How you read it is irrelevant to the truth :)
The matchlevel is per area, so there is no way to express multi-area match levels 'relative to the match level' - not in a sane way at least. But let's consider the normal case when every area has the default, 95% should be displayed as 95% and 97% as 97%, not a suprise jump. If the match rate was relative to the match level, then it should also be in the needle area display.
But that again would make it impossible to calculate a good match level. So KISS - and display 98% for a needle with one area matching 98%.
Updated by mkittler over 5 years ago
- Status changed from New to In Progress
- Target version changed from Ready to Current Sprint
I agree with @coolo. That's how I'm currently implementing it.
It looks like we don't store an "error" into the json for matched needles.
I've just imported that step result from o3 locally and added the "error" info which is indeed missing. That instantly let's the web UI show the percentage. So it is a bug/limitation only on the os-autoinst side.
Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?
Updated by okurz over 5 years ago
mkittler wrote:
Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?
I would go with "os-autoinst"