QA » openQA Project

Motivation¶

• It is less confusing to see the candidate needles how they actually were at the time the test run (AC1).
• Specifying Git repositories via CASEDIR/NEEDLES_DIR is made a first-class citizen and candidates needles are still shown at all when specifying a repository not checked on the web UI host.

Acceptance criteria¶

• AC1: The "diff view" on the test details pages shows the candidate needles (images and areas) how they were defined at the time the test run (when this feature is enabled in openqa.ini).
• AC2: The needle editor still allows to create new needles based on the latest version of existing needle images/areas.
• AC3: AC1 is also fulfilled if CASEDIR/NEEDLES_DIR points to a Git repository that is now checked out under $OPENQA_BASEDIR/openqa/share/tests.
  • Maybe we want to declare this AC optional as it complicates things. If it ends up not implemented as part of this ticket a follow-up ticket is created instead.
• AC4: If the ref cannot be shown (e.g. overridden after a force push) an indication that the latest version is shown instead should be displayed.
• AC5: If the ref can be shown an indication should be displayed so users are made aware of the new feature.
• AC6: Concurrent access and other usages of the Git repository (e.g. via fetchneedles) don't lead to errors or wrong needles being displayed.
• AC7: A follow-up ticket for improving the cleanup strategy has been created. (It has not been created yet because its details will highly depend on the outcome of this ticket.)

Suggestions¶

• See what has already been done as part on https://github.com/os-autoinst/openQA/pull/5175 as part of #154783.
  • Missing functionality for ACs
    • For AC2 the change needs to be adjusted so that the needle editor is only affected when enabled via an additional configuration.
    • For AC3 the code needs to check whether the repository needs to be cloned first and do that. Maybe we don't want to allow cloning arbitrary repositories, though.
    • For AC4 and AC5 it would make most sense to simply add a note on top of the image in the "diff view".
    • For AC6 the use of FETCH_HEAD needed to be avoided and maybe additional locking is required.
  • Further concerns
    • We might run into GitHub's rate limiting as this change always does a git fetch (except the ref is still cached).
    • It would cease to work for forks if GitHub stops allowing to fetch refs from forks by only using the original repository as remote. To avoid this we needed to add and use remotes as needed according to the Git URL in CASEDIR/NEEDLES_DIR. (But right now things should work for forks, see #154783#note-12.)
• UI tests need to be implemented, see https://github.com/Martchus/openQA/pull/new/needles_from_casedir_tests for a start.
• Consider recording the candidate needle images and area definitions when the test runs as part of the test result in case the approach from https://github.com/os-autoinst/openQA/pull/5175 doesn't work.

Out of scope¶

• Improve the cleanup
• Handle things sensibly within the needle editor
• Track not only the time but also the ref in the last seen/match statistics