action #49943

[tools] Add new needle match type: "click"

Added by dheidler about 1 year ago. Updated 11 months ago.

Status:ResolvedStart date:03/04/2019
Priority:NormalDue date:
Assignee:mkittler% Done:


Category:Feature requests
Target version:Current Sprint


Currently the testapi function assert_and_click is using the last area of a needle with type=match to click.
When there are multiple areas in a needle they can easily mixed up - especially as there is no indication in the needle editor
showing which area is the last one.

Therefore a new type of area with type=click should be added.
When such an area is present, only this area should be used for clicking.
Other than that it should behave like type=match areas.
When using assert_screen then there shouldn't be a difference between type=match and type=click areas.
A needle must not have more than one area with type=click - the needle testsuite will be adapted to fail otherwise.

Acceptance Criterias

  • AC1: The needle editor in the webui can handle such areas
    • Idea: Replace the unused (and also not even working) yellow ocr areas with click areas in the editor
  • AC2: In assert_and_click if there is a type=click area present in a needle then this area is used.
  • AC3: In assert_and_click if there is NO type=click area present in a needle the current behavior doesn't change for backwards compatibility.

Related issues

Duplicated by openQA Project - action #25712: feature request for assert_and_click - define exact area... Rejected 02/10/2017


#2 Updated by dheidler about 1 year ago

  • Description updated (diff)

#3 Updated by mkittler 12 months ago

The needle type so far determines how the match is computed. I'd rather not mess with that and instead add this as an additional flag. It would make more sense to show this with a mouse icon in the needle editor (instead of using a different color).

So far always the center of the needle is clicked, right? At least I remember people complaining about that limitation. Keeping that in mind I really opt for the "additional flag" and "mouse icon" solution because it could be extended more easily to also support relative click coordinates within the area.

#4 Updated by coolo 12 months ago

  • Category changed from 130 to 124
  • Target version set to Ready

It's still an area to be matched after all, so changing the type is wrong. We have needles where there are 2 intersecting areas in the hope that openQA would click on the smaller one. So defining a defined 'click point' sounds like the easier to scale option. If an area has a click point, it's the click area that this feature asks for. If not, we fall back to current logic.

#5 Updated by mkittler 12 months ago

  • Assignee set to mkittler
  • Target version changed from Ready to Current Sprint

#6 Updated by SLindoMansilla 12 months ago

Does it mean that each time a clickable area is added, the pixel position needs to be calculated? That would be painful.

Keep in mind that the main problem we encounter here is that when creating a new needle based on a old needle, the areas are cloned but in an arbitrary order, which causes the click area to be different. There should be a feature to select which area is to be clicked even when creating a new needle based on an old one.

So, the behavior of clicking on the center of the area is fine. Creating an extra little area to be the clickable one is easier to maintain than recalculating pixels.

I think mkittler's suggestion of an additional area-specific flag would be better than a click point

#7 Updated by mkittler 12 months ago

The click point can be easily added via the needle editor of course and will be by default the center of the area.

It should also be easy to allow both, click_point: {xpos: ..., ypos: ...} and click_point: "center", in case you create needles often without the help of the needle editor and don't want to calculate coordinates.

#9 Updated by mkittler 11 months ago

  • Status changed from In Progress to Feedback

PRs have been merged. The new needle editor UI for that feature could still be improved.

#10 Updated by dheidler 11 months ago

I really like this approach!

Just one thing that could be added:

It would be nice to (also?) use click_point coordinates relative to the containing match area.
This way it would be possible to handle different window positioning in different WMs using a single needle.
We could add an option for this in the needle editor and add some entry to click_point - eg: relative: true (should default to false if not present and omitted if not enabled).

#11 Updated by coolo 11 months ago

What else would the click point be? It's always relative, if it's not, it's a bug

#12 Updated by mkittler 11 months ago

We already talked about it, but for the record: Of course it is relative. Everything else makes not sense.

#13 Updated by mkittler 11 months ago

  • Status changed from Feedback to Resolved

Making this more fancy can be done when giving the needle editor a general overhaul. (But it is actually not that bad.)

#14 Updated by okurz 10 months ago

  • Duplicated by action #25712: feature request for assert_and_click - define exact area for click added

Also available in: Atom PDF