Project

General

Profile

action #109707

[qe-core] Detect when a maintenance update has been already released before running the test

Added by szarate 6 months ago. Updated 28 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Refactor/Code Improvements
Target version:
-
Start date:
2022-06-09
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Observation

To be refined, general idea is: when cloning or restarting a job, we should have a warning telling us that said job will fail, because the update has already been released

openQA test in scenario sle-15-SP3-Server-DVD-Updates-x86_64-mau-webserver+15@pablo-herranz/os-autoinst-distri-opensuse#php_new_version@64bit fails in
add_update_test_repo

Test suite description

Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml. SLE15+ version of mau-webserver

Reproducible

Fails since (at least) Build pablo-herranz/os-autoinst-distri-opensuse#php_new_version

Expected result

Last good: pablo-herranz/os-autoinst-distri-opensuse#php_new_version (or more recent)

Further details

Always latest result in this scenario: latest

Suggestions

  • If #109902 turns out to be a successful experiment, implemment something similar for this ticket.
  • Always remind to use the job
  • Check per incident, if repository in the settings are available before tests are running

Subtasks

action #112250: [QE-Core] zypper modify repo incorrect usage is muted in the patch_and_reboot testRejecteddvenkatachala


Related issues

Related to openQA Project - action #112349: Provide an option to always clone the latest version of the scenario run by the jobNew2022-06-13

Related to QA - action #111710: [qa-tools] [tools] remove usage of *_TEST_TEMPLATE vars in qem-bot and openQA media definition if favor of *_TEST_REPOS size:MResolved2022-05-272022-09-02

Related to openQA Project - action #116788: Cloning of job with released test issues not workingResolved2022-09-19

History

#1 Updated by szarate 6 months ago

  • Tags deleted (to-be-refined, qe-core-april-sprint)
  • Description updated (diff)

#3 Updated by szarate 5 months ago

  • Sprint set to QE-Core: May Sprint (May 11 - Jun 08)
  • Description updated (diff)
  • Start date deleted (2022-04-08)

Yet another example: https://suse.slack.com/archives/C02D16TCP99/p1650956217675289:

Job: https://openqa.suse.de/tests/8632058#step/patch_and_reboot/159 failed with:

2022-04-25 20:11:26 <5> linux(5692) [zypp] Exception.cc(log):166  - [TEST_24|http://dist.suse.de/ibs/SUSE:/Maintenance:/23830/SUSE_Updates_SLE-WSM_12-SP5_x86_64/] Repository type can't be determined.
2022-04-25 20:11:26 <5> linux(5692) [zypp] Exception.cc(log):166 
2022-04-25 20:11:26 <5> linux(5692) [zypp] Exception.cc(log):166 repos.cc(refresh_raw_metadata):395 CAUGHT:   [TEST_24|http://dist.suse.de/ibs/SUSE:/Maintenance:/23830/SUSE_Updates_SLE-WSM_12-SP5_x86_64/] Valid metadata not found at specified URL
2022-04-25 20:11:26 <5> linux(5692) [zypp] Exception.cc(log):166 History:
2022-04-25 20:11:26 <5> linux(5692) [zypp] Exception.cc(log):166  - [TEST_24|http://dist.suse.de/ibs/SUSE:/Maintenance:/23830/SUSE_Updates_SLE-WSM_12-SP5_x86_64/] Repository type can't be determined.

#4 Updated by szarate 5 months ago

  • Status changed from New to Blocked

#5 Updated by dvenkatachala 4 months ago

  • Status changed from Blocked to Workable

Anna and I had discussion, she suggested that we can implement the changes in clone_job.pl and openqa-clone-custom-git-refspec.
I will make the change as per suggestion.

#6 Updated by dvenkatachala 4 months ago

  • Assignee set to dvenkatachala

#7 Updated by dvenkatachala 4 months ago

  • Status changed from Workable to In Progress

#8 Updated by szarate 4 months ago

Dee to update the ticket

#9 Updated by dvenkatachala 4 months ago

I have done following changes in Script/CloneJob.pm

  1. Added a new module to verify the maintenance updates.
  2. In this module, defined a hash: the key (_TEST_TEMPLATE) => corresponding value (_TEST_ISSUE).
  3. Later this module substitutes the "@INCIDENTNR@" of TEST_TEMPLATE URLs with each corresponding TEST_ISSUE (defined in job settings) to verify the maintenance updates.

Different scenario:
In a few job settings templates, for e.g, https://openqa.suse.de/tests/8911139#settings . I see TEST_TEMPLATE with no associated incidents. How can I address such settings? Do you have any suggestions?

Further observation:
While reviewing this job(https://openqa.suse.de/tests/8911139) and I noticed a false positive in the test : https://openqa.suse.de/tests/8911139#step/patch_and_reboot/15

#10 Updated by szarate 4 months ago

While we wait for an agreement on https://github.com/os-autoinst/openQA/pull/4693

dvenkatachala wrote:

I have done following changes in Script/CloneJob.pm

  1. Added a new module to verify the maintenance updates.
  2. In this module, defined a hash: the key (_TEST_TEMPLATE) => corresponding value (_TEST_ISSUE).
  3. Later this module substitutes the "@INCIDENTNR@" of TEST_TEMPLATE URLs with each corresponding TEST_ISSUE (defined in job settings) to verify the maintenance updates.

Different scenario:
In a few job settings templates, for e.g, https://openqa.suse.de/tests/8911139#settings . I see TEST_TEMPLATE with no associated incidents. How can I address such settings? Do you have any suggestions?

I think the logic here could go towards getting all the keys using a regex, so anything that matches _TEST_TEMPLATE should have a _TEST_ISSUES pair, so in this case we have LTSS... using a regex or grepping would help in the future to automatically add new products

Further observation:
While reviewing this job(https://openqa.suse.de/tests/8911139) and I noticed a false positive in the test : https://openqa.suse.de/tests/8911139#step/patch_and_reboot/15

Dee, could you catch this particular case?

#13 Updated by szarate 4 months ago

  • Sprint changed from QE-Core: May Sprint (May 11 - Jun 08) to QE-Core: June Sprint (Jun 08 - Jul 06)

#14 Updated by dvenkatachala 4 months ago

  • Tags changed from qe-core-april-sprint, qe-core-may-sprint, qe-core-june-sprint to qe-core-april-sprint, qe-core-may-sprint

szarate wrote:

While we wait for an agreement on https://github.com/os-autoinst/openQA/pull/4693

dvenkatachala wrote:

I have done following changes in Script/CloneJob.pm

  1. Added a new module to verify the maintenance updates.
  2. In this module, defined a hash: the key (_TEST_TEMPLATE) => corresponding value (_TEST_ISSUE).
  3. Later this module substitutes the "@INCIDENTNR@" of TEST_TEMPLATE URLs with each corresponding TEST_ISSUE (defined in job settings) to verify the maintenance updates.

Different scenario:
In a few job settings templates, for e.g, https://openqa.suse.de/tests/8911139#settings . I see TEST_TEMPLATE with no associated incidents. How can I address such settings? Do you have any suggestions?

I think the logic here could go towards getting all the keys using a regex, so anything that matches _TEST_TEMPLATE should have a _TEST_ISSUES pair, so in this case we have LTSS... using a regex or grepping would help in the future to automatically add new products

Further observation:
While reviewing this job(https://openqa.suse.de/tests/8911139) and I noticed a false positive in the test : https://openqa.suse.de/tests/8911139#step/patch_and_reboot/15

Dee, could you catch this particular case?

Santi, I am sorry I made wrong analysis here. There is no issue in the test.

#15 Updated by szarate 3 months ago

  • Related to action #112349: Provide an option to always clone the latest version of the scenario run by the job added

#16 Updated by szarate 3 months ago

Hi!, can you provide an update of the current state of the ticket?

#17 Updated by dvenkatachala 3 months ago

I could not proceed much due to Hack week and review shifts. I will continue working on this ticket after review/bugbuster.

#18 Updated by szarate 3 months ago

  • Sprint changed from QE-Core: June Sprint (Jun 08 - Jul 06) to QE-Core: July Sprint (Jul 06 - Aug 03)

#19 Updated by dvenkatachala 2 months ago

  • Related to action #111710: [qa-tools] [tools] remove usage of *_TEST_TEMPLATE vars in qem-bot and openQA media definition if favor of *_TEST_REPOS size:M added

#20 Updated by dvenkatachala 2 months ago

The PR 4693 verifies the maintenance updates availability by constructing the Template_URL from the test variables _TEST_TEMPLATE and corresponding _TEST_ISSUES pair.

Next step is to look for the MAINT_TEST_REPO instead of the _TEST_TEMPLATE from the test variable to verify the maintenance updates after the implementation of the poo#111710.

#21 Updated by szarate about 2 months ago

  • Sprint changed from QE-Core: July Sprint (Jul 06 - Aug 03) to QE-Core: August Sprint (Aug 03 - Aug 31)

#22 Updated by dvenkatachala about 1 month ago

Output snippets from the verification runs,
Aggregated jobs:

  1. Current job 9289622 will fail, because the repositories for the below updates are unavailable [ "https://download.suse.de/ibs/SUSE:/Maintenance:/25166/SUSE_Updates_SLE-Product-SLES_15-LTSS_x86_64/", "https://download.suse.de/ibs/SUSE:/Maintenance:/25296/SUSE_Updates_SLE-Product-SLES_15-LTSS_x86_64/", ] at /usr/share/openQA/script/../lib/OpenQA/Script/CloneJobSUSE.pm line 37.

2.Current job 9289071 will fail, because the repositories for the below updates are unavailable
[
"https://download.suse.de/ibs/SUSE:/Maintenance:/24448/SUSE_Updates_SLE-Module-SAP-Applications_15-SP2_x86_64/",
"https://download.suse.de/ibs/SUSE:/Maintenance:/25299/SUSE_Updates_SLE-Module-SAP-Applications_15-SP2_x86_64/",
] at /usr/share/openQA/script/../lib/OpenQA/Script/CloneJobSUSE.pm line 37.

Incident Jobs:

  1. Current job 9223278 will fail, because the repositories for the below updates are unavailable
    [
    "http://download.suse.de/ibs/SUSE:/Maintenance:/25210/SUSE_Updates_SLE-Product-SLES_15-LTSS_x86_64",
    ] at /usr/share/openQA/script/../lib/OpenQA/Script/CloneJobSUSE.pm line 37.

  2. Current job 9109827 will fail, because the repositories for the below updates are unavailable
    [
    "http://download.suse.de/ibs/SUSE:/Maintenance:/24981/SUSE_Updates_SLE-Product-SLES_15-SP1-LTSS_x86_64",
    ] at /usr/share/openQA/script/../lib/OpenQA/Script/CloneJobSUSE.pm line 37.

#23 Updated by dvenkatachala 28 days ago

  • Status changed from In Progress to Resolved

#24 Updated by okurz 7 days ago

  • Related to action #116788: Cloning of job with released test issues not working added

Also available in: Atom PDF