Project

General

Profile

Actions

action #32725

closed

coordination #32851: [tools][EPIC] Scheduling redesign

[tools] Scheduler job_grab/filter_jobs refactoring

Added by EDiGiacinto about 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2018-05-05
Due date:
% Done:

0%

Estimated time:

Description

For the moment it would be sufficient to refactor job_grab, and pass over schemas objects instead of hashes and re-use them in the filtering phase, meanwhile refactoring properly those areas (e.g. extracting them) - better if we also optimize the queries.

Kinda of a hard task, since could potentially make things 'shaky'.


Related issues 2 (0 open2 closed)

Related to openQA Project - action #39560: Tests for blocked_by and loops inside of itResolved2018-08-10

Actions
Related to openQA Project - action #39629: openQA Scheduler refactor falloutResolved2018-08-13

Actions
Actions #1

Updated by EDiGiacinto about 6 years ago

  • Description updated (diff)
Actions #2

Updated by EDiGiacinto about 6 years ago

  • Description updated (diff)
Actions #3

Updated by coolo about 6 years ago

  • Target version set to Ready

Taken that we usually have < 2000 jobs scheduled, I think it's worth an experiment to grab all open jobs and schedule in memory. There is only one scheduler process, so IMO this should simplify a lot. Yes, hard task - but worth the time spent to make the whole scheduling understandable.

Actions #4

Updated by EDiGiacinto about 6 years ago

  • Description updated (diff)
  • Parent task set to #32851
Actions #5

Updated by szarate almost 6 years ago

  • Start date changed from 2018-03-02 to 2018-05-05

due to changes in a related task

Actions #6

Updated by szarate over 5 years ago

  • Target version changed from Ready to Current Sprint

We agreed during the sprint planning to keep going with this: https://github.com/os-autoinst/openQA/pull/1718 ettore will take over coolo's branch.

Actions #7

Updated by EDiGiacinto over 5 years ago

PR: https://github.com/os-autoinst/openQA/pull/1729 ( missing acceptance testing now )

Actions #8

Updated by szarate over 5 years ago

  • Assignee set to EDiGiacinto

I think there are still some rough edges here, right?

Actions #9

Updated by EDiGiacinto over 5 years ago

  • Status changed from New to Feedback
  • Assignee deleted (EDiGiacinto)

Yes, but as i'm not the author of this set of changes, can't help much more from here - i adapted the tests we already had for the scheduler and did some adaptations to the issues i could notice, but i think there are still bugs to catch.

Latest related change to it is: https://github.com/os-autoinst/openQA/pull/1743

Actions #10

Updated by EDiGiacinto over 5 years ago

  • Related to action #39560: Tests for blocked_by and loops inside of it added
Actions #11

Updated by EDiGiacinto over 5 years ago

  • Related to action #39629: openQA Scheduler refactor fallout added
Actions #12

Updated by coolo over 5 years ago

  • Status changed from Feedback to Resolved

The scheduler was refactored - we have some new problems (like starving multimachine jobs), but they are design decisions and need future fine tuning

Actions #13

Updated by coolo over 5 years ago

  • Target version changed from Current Sprint to Done
Actions

Also available in: Atom PDF