action #45746
closed
Needle selection shows 100% for matched needles (even if 98%)
Added by zgao over 5 years ago.
Updated over 5 years ago.
Category:
Regressions/Crashes
Description
In this case on o3, the needle seems to be matched at similarity 98%, but the dropdown shows a similarity of 100% ---- which is not correct.
I thought of the similarity threshold of needles but in this needle, gnome-screenlock-password-Tumbleweed-20181012, no similarity threshold has been specified. So what is the default threshold for needle matching?
I presume it's a openQA bug, what do you think?
As zcjia kindly points out, default similarity threshold seems to be 96% as source code shows.
Can we say it is buggy when needles shows similarity 98% while dropdown shows similarity 100%?
- Subject changed from Needle falsely matched 100% on o.o.o to Needle selection shows 100% for matched needles (even if 98%)
- Category set to Regressions/Crashes
- Target version set to Ready
- Difficulty set to easy
It looks like we don't store an "error" into the json for matched needles. But this is used to calculcate the percentage. So no error means 0, means 100%.
And how do you plan to solve this?
I read the value in the pull down menu as: "Matches 100% relative to the match level or higher". So if you have a match level of 96% as per default then there can be an actual, absolute match of 96% up to 100% all considered as "passed"
How you read it is irrelevant to the truth :)
The matchlevel is per area, so there is no way to express multi-area match levels 'relative to the match level' - not in a sane way at least. But let's consider the normal case when every area has the default, 95% should be displayed as 95% and 97% as 97%, not a suprise jump. If the match rate was relative to the match level, then it should also be in the needle area display.
But that again would make it impossible to calculate a good match level. So KISS - and display 98% for a needle with one area matching 98%.
- Status changed from New to In Progress
- Target version changed from Ready to Current Sprint
I agree with @coolo. That's how I'm currently implementing it.
It looks like we don't store an "error" into the json for matched needles.
I've just imported that step result from o3 locally and added the "error" info which is indeed missing. That instantly let's the web UI show the percentage. So it is a bug/limitation only on the os-autoinst side.
Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?
mkittler wrote:
Btw, I've started to compose at least a little bit documentation for the result format. When this is done, to which repo would I add it? Is the result format defined on the os-autoinst side or on the openQA side?
I would go with "os-autoinst"
- Status changed from In Progress to Resolved
Also available in: Atom
PDF