Project

General

Profile

action #76747

Define exclude areas relative to match areas (not absolute position) (was: Needle masks do not handle matches with offsets)

Added by StefanBruens 9 months ago. Updated 5 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Original Motivation

Observation

openQA test in scenario opensuse-Tumbleweed-KDE-Live-x86_64-kde-live@64bit fails in
shutdown

The match is positioned correctly, but the masks are applied at the original location.

Reproducible

Fails since (at least) Build 20201027 (current job)

Expected result

Last good: 20201026 (or more recent)

Further details

Always latest result in this scenario: latest

Acceptance criteria

  • AC1: Exclude areas are defined relative to the surrounding match area(s)

History

#1 Updated by tjyrinki_suse 8 months ago

  • Subject changed from Needle masks do not handle matches with offsets to [qe-core][opensuse]Needle masks do not handle matches with offsets
  • Status changed from New to Resolved

This looks like not happening anymore in shutdown phase so probably needles were fixed.

#2 Updated by StefanBruens 8 months ago

  • Status changed from Resolved to New

This is not about some specific needles, but a generic deficiency in the implementation.

#3 Updated by tjyrinki_suse 8 months ago

  • Status changed from New to Workable
  • Start date deleted (2020-10-28)

#4 Updated by tjyrinki_suse 8 months ago

  • Status changed from Workable to New

#5 Updated by tjyrinki_suse 5 months ago

  • Subject changed from [qe-core][opensuse]Needle masks do not handle matches with offsets to [tools] Needle masks do not handle matches with offsets

The generic needle offset handling something to be evaluated by the openQA developers.

#6 Updated by okurz 5 months ago

  • Target version set to Ready

#7 Updated by mkittler 5 months ago

This is not about some specific needles, but a generic deficiency in the implementation.

Than it would be great to come up with steps to reproduce and expected vs. actual behavior. Note that at this point not even the original job is available anymore so I have really no idea for what kind of improvement you're asking. By the way, what exactly do you mean with "needle masks"?

#8 Updated by okurz 5 months ago

mkittler wrote:

By the way, what exactly do you mean with "needle masks"?

I assume the needle match areas with defined size and also the initially defined position where IIRC we also start a search for a match.

#9 Updated by StefanBruens 5 months ago

okurz wrote:

mkittler wrote:

By the way, what exactly do you mean with "needle masks"?

I assume the needle match areas with defined size and also the initially defined position where IIRC we also start a search for a match.

I am referring to "excluded" match areas. See needle.pm in os-autoinst.

#10 Updated by okurz 5 months ago

  • Status changed from New to Feedback
  • Assignee set to okurz

oh, so you want to have exclude areas to be defined relative to the match areas rather than absolute areas?

#11 Updated by StefanBruens 5 months ago

okurz wrote:

oh, so you want to have exclude areas to be defined relative to the match areas rather than absolute areas?

I want the exclude areas to be consistent with the match areas. Everything else doesn't make sense and even creates false matches.

Imagine an entry in a list, where the icon in front of the label is excluded. A new entry is added in a new version, and the original entry moves down by one. The exclude area now overpaints the (excluded) icon of the entry above the real match with a blue rectangle, and also the icon of the needle area (that how excluding currently works). This has the effect the old position becomes a fairly good match, and the new position is a bad match because nothing is excluded there (even worse, made significantly different). Combined with the preference of small match offsets and weighting of the error by needle area size, at best we get no sufficient match at all, or at worst a false match.

#12 Updated by okurz 5 months ago

  • Project changed from openQA Tests to openQA Project
  • Subject changed from [tools] Needle masks do not handle matches with offsets to Define exclude areas relative to match areas (not absolute position) (was: Needle masks do not handle matches with offsets)
  • Description updated (diff)
  • Category changed from Bugs in existing tests to Feature requests
  • Status changed from Feedback to New
  • Assignee deleted (okurz)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

Well, at what you describe sounds sensible but we have to understand that this would be a breaking change. The current behaviour might be badly described or not at all but this is how it currently works. Having the exclude relative to match areas sounds useful at least as an configurable alternative. We have to see what we can do eventually. Converted to feature request for openQA

#13 Updated by mkittler 5 months ago

Having the exclude relative to match areas sounds useful at least as an configurable alternative.

Yes, it could be a property of the exclude area.

I assume the exclude area would be tied to the match area it intersects. The raises the question what should happen if an exclude area intersects multiple match areas. Maybe we better allow the user to tie an exclude area manually to a match area within the needle editor?

By the way, the exclude areas are currently implemented by simply filling the parts of the images with a green rectangle. I'm not sure how complicated it'll be to make it more dynamic.

#14 Updated by coolo 5 months ago

The needle matching is happening in 2 phases: find the pattern and then judge the percentage. You can in the middle of it blank parts of the image. That's generally quite expensive, but doable.
But if that area to be excluded breaks your pattern matching, you're screwed. So don't expect miracles. This will only work for relatively large areas where the area to be excluded is rather a nuance.

#15 Updated by StefanBruens 5 months ago

coolo wrote:

The needle matching is happening in 2 phases: find the pattern and then judge the percentage. You can in the middle of it blank parts of the image. That's generally quite expensive, but doable.
But if that area to be excluded breaks your pattern matching, you're screwed. So don't expect miracles. This will only work for relatively large areas where the area to be excluded is rather a nuance.

You can do matching with "moving" exclude areas in a comparatively cheap way. As error used for finding the best location is a linear sum over the area, you can calculate the error of on arbitrarily shaped area by adding/subtracting smaller rectangular areas. I.e., Err_excluded = Err_full - Err_excludearea.

The addition/subtraction in general is trivial (algorithmically and computationally, relative to the matching itself). Some care is necessary when the positions are restricted e.g. when the match area is near the screen border, we have to clamp the error matrices so all use the same offsets.

I think I can write up some opencv POC over the weekend and provide some test cases.

#16 Updated by coolo 5 months ago

That means duplicating the template matching for every excluded region per needle. E_full-E_area might be cheap (in comparision), but calculating E_erea is not. And if I imagine that this becomes popular, we will get even more of "why is needle matching so slow?

I'm not sure the price is worth it.

#17 Updated by StefanBruens 5 months ago

coolo wrote:

That means duplicating the template matching for every excluded region per needle. E_full-E_area might be cheap (in comparision), but calculating E_erea is not. And if I imagine that this becomes popular, we will get even more of "why is needle matching so slow?

I'm not sure the price is worth it.

It depends. We already have cases where people (ab)use multiple areas. One case I am aware of is the matching of "#" for the console tests, which uses one area expanding to the right, and one area expanding down. Handling these together will actually be (slightly) cheaper.

And there is also the possibility to make the matching faster. Very often, we match N needles over M frames. Some of the calculations happening inside openCVs template_match are invariant in respect to the needle or the haystack, so instead of doing these MxN times, we can do these only N resp. M times. (Examples: DFT as preparation for the fast convolution, Integral images for error calculation). Currently, we do the invariant preprocessing only for the part happening outside openCV, e.g. color-to-gray conversion, blurring.

Also available in: Atom PDF