action #136136: [qe-core] jobs scheduled for currently not available worker31 "ovs-server+client" - openQA Tests (public) - openSUSE Project Management Tool

Custom queries

All open Feature tests
openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QAM
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
SLE15 Migration Open Tickets
SLE15 SP1 Migration Open Tickets
SLE15SP3 Migration open ticket
SLE15SP3 Security open ticket
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #136136

closed

[qe-core] jobs scheduled for currently not available worker31 "ovs-server+client"

Added by okurz over 1 year ago. Updated about 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

dzedro

Category:

Bugs in existing tests

Target version:

Start date:

2023-09-20

Due date:

% Done:

Estimated time:

Difficulty:

Description

Observation¶

https://openqa.suse.de/tests/12196999# "ovs-server" is scheduled for WORKER_CLASS combination "qemu_x86_64,tap,worker31" which is currently not fulfillable because the specific machine is not available as an openQA worker. The worker specific pinning needs to be removed.

Related issues 1 (0 open — 1 closed)

Related to openQA Tests (public) - action #135818: [kernel] minimal reproducer for many multi-machine test failures in "ovs-client+ovs-server" test scenario when tests are run across different workers

Resolved

pcervinka

2023-08-15

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by okurz over 1 year ago

Subject changed from jobs scheduled for currently not available worker31 "ovs-server+client" to [qe-core] jobs scheduled for currently not available worker31 "ovs-server+client"

Actions

Copy link

Updated by okurz over 1 year ago

Related to action #135818: [kernel] minimal reproducer for many multi-machine test failures in "ovs-client+ovs-server" test scenario when tests are run across different workers added

Actions

Copy link

Updated by okurz over 1 year ago

Priority changed from Urgent to High

To handle the currently scheduled stuck jobs I did

for i in $(openqa-cli api --osd jobs state=scheduled | jq -r '.jobs | .[] | select(.settings.WORKER_CLASS|match("worker31")) | .id'); do openqa-cli api --osd -X post jobs/$i/comments text="poo#136136" && openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/$i WORKER_CLASS="qemu_x86_64,tap"; done

yielding

{"id":1033271}
Cloning parents of sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit
Cloning children of sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit
Cloning parents of sle-15-SP6-Online-x86_64-Build20.1-ovs-client@64bit
2 jobs have been created:
 - sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit -> https://openqa.suse.de/tests/12207581
 - sle-15-SP6-Online-x86_64-Build20.1-ovs-client@64bit -> https://openqa.suse.de/tests/12207582
{"id":1033272}
Cloning parents of sle-15-SP6-Online-x86_64-Build20.1-ovs-client@64bit
Cloning parents of sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit
Cloning children of sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit
2 jobs have been created:
 - sle-15-SP6-Online-x86_64-Build20.1-ovs-server@64bit -> https://openqa.suse.de/tests/12207584
 - sle-15-SP6-Online-x86_64-Build20.1-ovs-client@64bit -> https://openqa.suse.de/tests/12207583
{"id":1033273}
Cloning parents of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning parents of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
2 jobs have been created:
 - sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207585
 - sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207586
{"id":1033274}
Cloning parents of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
Cloning parents of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
2 jobs have been created:
 - sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207587
 - sle-15-SP3-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207588
{"id":1033275}
Cloning parents of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning parents of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
2 jobs have been created:
 - sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207590
 - sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207589
{"id":1033276}
Cloning parents of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
Cloning parents of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
2 jobs have been created:
 - sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207592
 - sle-15-SP4-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207591
{"id":1033277}
Cloning parents of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning parents of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
2 jobs have been created:
 - sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207594
 - sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207593
{"id":1033278}
Cloning parents of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
Cloning parents of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
Cloning children of sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit
2 jobs have been created:
 - sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-server@64bit -> https://openqa.suse.de/tests/12207602
 - sle-15-SP5-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207601
{"id":1033279}
Cloning parents of sle-15-SP2-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit
1 job has been created:
 - sle-15-SP2-Server-DVD-Updates-x86_64-Build20230919-1-ovs-client@64bit -> https://openqa.suse.de/tests/12207609

Actions

Copy link

Updated by okurz over 1 year ago

Still found scheduled jobs blocking SLE maintenance releases, e.g. https://openqa.suse.de/tests/12228008

Actions

Copy link

Updated by okurz over 1 year ago

I have now removed the worker class setting for worker31 in https://openqa.suse.de/admin/test_suites

For all currently stuck jobs from https://openqa.suse.de/tests?match=ovs-server I have executed

for i in 12238298 12237916 12237682 12237487; do openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/$i WORKER_CLASS=qemu_x86_64,tap; done

and labeled and cancelled the according old jobs

Actions

Copy link