action #76747
openDefine exclude areas relative to match areas (not absolute position) (was: Needle masks do not handle matches with offsets)
0%
Description
Original Motivation¶
Observation¶
openQA test in scenario opensuse-Tumbleweed-KDE-Live-x86_64-kde-live@64bit fails in
shutdown
The match is positioned correctly, but the masks are applied at the original location.
Reproducible¶
Fails since (at least) Build 20201027 (current job)
Expected result¶
Last good: 20201026 (or more recent)
Further details¶
Always latest result in this scenario: latest
Acceptance criteria¶
- AC1: Exclude areas are defined relative to the surrounding match area(s)
Updated by tjyrinki_suse about 4 years ago
- Subject changed from Needle masks do not handle matches with offsets to [qe-core][opensuse]Needle masks do not handle matches with offsets
- Status changed from New to Resolved
This looks like not happening anymore in shutdown phase so probably needles were fixed.
Updated by StefanBruens about 4 years ago
- Status changed from Resolved to New
This is not about some specific needles, but a generic deficiency in the implementation.
Updated by tjyrinki_suse about 4 years ago
- Status changed from New to Workable
- Start date deleted (
2020-10-28)
Updated by tjyrinki_suse almost 4 years ago
- Subject changed from [qe-core][opensuse]Needle masks do not handle matches with offsets to [tools] Needle masks do not handle matches with offsets
The generic needle offset handling something to be evaluated by the openQA developers.
Updated by mkittler over 3 years ago
This is not about some specific needles, but a generic deficiency in the implementation.
Than it would be great to come up with steps to reproduce and expected vs. actual behavior. Note that at this point not even the original job is available anymore so I have really no idea for what kind of improvement you're asking. By the way, what exactly do you mean with "needle masks"?
Updated by okurz over 3 years ago
mkittler wrote:
By the way, what exactly do you mean with "needle masks"?
I assume the needle match areas with defined size and also the initially defined position where IIRC we also start a search for a match.
Updated by StefanBruens over 3 years ago
okurz wrote:
mkittler wrote:
By the way, what exactly do you mean with "needle masks"?
I assume the needle match areas with defined size and also the initially defined position where IIRC we also start a search for a match.
I am referring to "excluded" match areas. See needle.pm in os-autoinst.
Updated by okurz over 3 years ago
- Status changed from New to Feedback
- Assignee set to okurz
oh, so you want to have exclude areas to be defined relative to the match areas rather than absolute areas?
Updated by StefanBruens over 3 years ago
okurz wrote:
oh, so you want to have exclude areas to be defined relative to the match areas rather than absolute areas?
I want the exclude areas to be consistent with the match areas. Everything else doesn't make sense and even creates false matches.
Imagine an entry in a list, where the icon in front of the label is excluded. A new entry is added in a new version, and the original entry moves down by one. The exclude area now overpaints the (excluded) icon of the entry above the real match with a blue rectangle, and also the icon of the needle area (that how excluding currently works). This has the effect the old position becomes a fairly good match, and the new position is a bad match because nothing is excluded there (even worse, made significantly different). Combined with the preference of small match offsets and weighting of the error by needle area size, at best we get no sufficient match at all, or at worst a false match.
Updated by okurz over 3 years ago
- Project changed from openQA Tests to openQA Project
- Subject changed from [tools] Needle masks do not handle matches with offsets to Define exclude areas relative to match areas (not absolute position) (was: Needle masks do not handle matches with offsets)
- Description updated (diff)
- Category changed from Bugs in existing tests to Feature requests
- Status changed from Feedback to New
- Assignee deleted (
okurz) - Priority changed from Normal to Low
- Target version changed from Ready to future
Well, at what you describe sounds sensible but we have to understand that this would be a breaking change. The current behaviour might be badly described or not at all but this is how it currently works. Having the exclude relative to match areas sounds useful at least as an configurable alternative. We have to see what we can do eventually. Converted to feature request for openQA
Updated by mkittler over 3 years ago
Having the exclude relative to match areas sounds useful at least as an configurable alternative.
Yes, it could be a property of the exclude area.
I assume the exclude area would be tied to the match area it intersects. The raises the question what should happen if an exclude area intersects multiple match areas. Maybe we better allow the user to tie an exclude area manually to a match area within the needle editor?
By the way, the exclude areas are currently implemented by simply filling the parts of the images with a green rectangle. I'm not sure how complicated it'll be to make it more dynamic.
Updated by coolo over 3 years ago
The needle matching is happening in 2 phases: find the pattern and then judge the percentage. You can in the middle of it blank parts of the image. That's generally quite expensive, but doable.
But if that area to be excluded breaks your pattern matching, you're screwed. So don't expect miracles. This will only work for relatively large areas where the area to be excluded is rather a nuance.
Updated by StefanBruens over 3 years ago
coolo wrote:
The needle matching is happening in 2 phases: find the pattern and then judge the percentage. You can in the middle of it blank parts of the image. That's generally quite expensive, but doable.
But if that area to be excluded breaks your pattern matching, you're screwed. So don't expect miracles. This will only work for relatively large areas where the area to be excluded is rather a nuance.
You can do matching with "moving" exclude areas in a comparatively cheap way. As error used for finding the best location is a linear sum over the area, you can calculate the error of on arbitrarily shaped area by adding/subtracting smaller rectangular areas. I.e., Err_excluded = Err_full - Err_excludearea.
The addition/subtraction in general is trivial (algorithmically and computationally, relative to the matching itself). Some care is necessary when the positions are restricted e.g. when the match area is near the screen border, we have to clamp the error matrices so all use the same offsets.
I think I can write up some opencv POC over the weekend and provide some test cases.
Updated by coolo over 3 years ago
That means duplicating the template matching for every excluded region per needle. E_full-E_area might be cheap (in comparision), but calculating E_erea is not. And if I imagine that this becomes popular, we will get even more of "why is needle matching so slow?
I'm not sure the price is worth it.
Updated by StefanBruens over 3 years ago
coolo wrote:
That means duplicating the template matching for every excluded region per needle. E_full-E_area might be cheap (in comparision), but calculating E_erea is not. And if I imagine that this becomes popular, we will get even more of "why is needle matching so slow?
I'm not sure the price is worth it.
It depends. We already have cases where people (ab)use multiple areas. One case I am aware of is the matching of "#" for the console tests, which uses one area expanding to the right, and one area expanding down. Handling these together will actually be (slightly) cheaper.
And there is also the possibility to make the matching faster. Very often, we match N needles over M frames. Some of the calculations happening inside openCVs template_match are invariant in respect to the needle or the haystack, so instead of doing these MxN times, we can do these only N resp. M times. (Examples: DFT as preparation for the fast convolution, Integral images for error calculation). Currently, we do the invariant preprocessing only for the part happening outside openCV, e.g. color-to-gray conversion, blurring.