coordination #32851: [tools][EPIC] Scheduling redesign
[epic] Offer a way for jobs to dynamically schedule children
As discussed with Vladimir, we want to offer an API in mmapi for tests to schedule children. The use case are mainly slenkins control nodes, but we can also use it to ease dynamic tests.
The use case is:
slenkins-nfs-control starts and installs the slenkins package into the VM.
The test then figures from the given control file that it needs a nfs-server and a nfs-client with a
given set of settings (currently implemented in the slenkins-import script).
It asks mmapi to schedule two tests as children following its own settings, but with a set of new variables (each).
If mmapi refuses to schedule them, the test waits and retries in a couple of minutes.
If the jobs are scheduled, we get to the normal mmapi function to wait for the children to boot before we call into slenkins.
openQA in return will schedule the children jobs with a high priority (to avoid dead locks), but will refuse to create
any jobs if there are still scheduled jobs with that high priority (or some other means to check other children are taken).
And we need to make sure the slenkins tests are only run on a very limited amount of workers - as they fork children basically
the formula is number(slenkins tests) = number(available nodes with slenkins worker class) - max(possible children). Otherwise
we can create a deadlock.
The benefit is that slenkins tests don't require manual mantenance anymore, but only once registration.
#1 Updated by nadvornik over 4 years ago
some additional ideas since yesterday:
mmapi tries to schedule jobs, it fails, the test tries again after some time
alternative to this: scheduling jobs will never fail, the jobs will appear as "scheduled", scheduler decides when to run them.
Scheduler starts the parallel test groups one after another, the situation when multiple groups are partially started can't happen and thus it is not handled currently.
The problem with parsing node file is that the job that parses it and creates other jobs is part of a group, but the scheduler does not know it from the beginning.
1 Improve scheduler to handle the situation when the first job from each group is already running and start the remaining jobs one group after another. This leads to the limitation number(slenkins tests) = number(available nodes with slenkins worker class) - max(possible children).
2 Make the job that parses the node file independent of the jobs it creates. All node files can be actually parsed in one job. This is compatible with current scheduler.
3 Number(slenkins tests) = number(available nodes with slenkins worker class) / max(possible children)
- this is most conservative limitation, no deadlock is possible
- wastes resources, unless there is enough single-machine jobs to run on the remaining workers example: 20 workers, max 8 children per test, typically 3, number of slenkins tests = 2, typical load 2 * 3 = 6 workers of 20
In any case, the jobs in a group should be created atomically, current api does not allow this. I think that clone_job.pl also suffers from this problem.
- create multiple jobs in one api call
- introduce new status "prepared", create jobs with this status, add api call to atomically switch whole group from prepared to scheduled
#3 Updated by coolo about 3 years ago
- Subject changed from Offer a way for jobs to schedule children to [epic] Offer a way for jobs to schedule children
- Status changed from In Progress to New
- Assignee deleted (
- Priority changed from Normal to Low
- Target version set to Ready
This is still very much wanted for more comfort in multi machine tests.
"Epic" is a good category here :-)
Jokes aside: okurz: is this still valid ticket? we do not run SLEnkins test in openQA, right? Or am I miss something?
"The use case are mainly slenkins control nodes"
Am I correct that in the meantime some of the ideas got implemented?
I would be quite interested in refining this ticket in technical details and see what we could do about it, as we both agree (I assume) mm tests could benefit from some enhancements or perhaps there could be completely new take on them where we would end up with another way of doing mm test in openQA?
- Status changed from New to Rejected
- Assignee set to okurz
True, SLEnkins test cases do not seem to be relevant nowadays. The ticket here is about dynamically scheduling children, i.e. define tests from already running test code. I have noted down the idea from this ticket now in the generalized feature request tracker #65271 so we can close this one here.
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html