action #91782
closedcoordination #64746: [saga][epic] Scale up: Efficient handling of large storage to be able to run current tests efficiently but keep big archives of old results
coordination #80546: [epic] Scale up: Enable to store more results
Add support for archived jobs
Description
Motivation¶
See parent #80546 . To be able to scale-up openQA we want to be able to separate recent openQA results (on expensive+fast storage) from older, important openQA results (on slow+cheap storage).
Acceptance criteria¶
- AC1: results on openQA can be automatically separated into two different storage targets
- AC2: https://github.com/os-autoinst/openQA/pull/3858 is merged or closed
Suggestions¶
- Add help in web UI based on https://github.com/os-autoinst/openQA/pull/3858 describing the impact of periods selected for "important" jobs when archiving is enabled
- Add comments in the example openqa.ini describing the archiving feature based on https://github.com/os-autoinst/openQA/pull/3858
- Add documentation for the new openQA feature based on https://github.com/os-autoinst/openQA/pull/3858 where the internal help and the comments in openqa.ini seem not enough
- Run tests, e.g. in SUSE internal's staging setup
- Add a complete implementation of the draft https://github.com/os-autoinst/openQA/pull/3858 - continue and finish it or start over :)
Out of scope¶
- UI representation for archived jobs: #91785
Updated by mkittler about 4 years ago
Since I've already started with https://github.com/os-autoinst/openQA/pull/3858 I'd also like to finish it. So I'll take the ticket when it is moved to the backlog.
Updated by okurz about 4 years ago
- Due date set to 2021-05-25
- Status changed from Workable to In Progress
Updated by mkittler almost 4 years ago
- Status changed from In Progress to Feedback
I'm waiting for PR reviews. See the commit message of the last commit for remaining caveats. For now I'd refrain from any complicated hacks to overcome them as they shouldn't be very problematic in practice.
Updated by okurz almost 4 years ago
- Related to coordination #76984: [epic] Automatically remove assets+results based on available free space added
Updated by mkittler almost 4 years ago
- Status changed from Feedback to Resolved
I'm closing the ticket despite the remaining caveats. The archiving job takes care to delete any leftovers if copying fails so if cleanup and archiving interfere with each other the worst which can happen is a failed archiving job (but leftovers should not pile up in the archive). Besides, this is very unlikely anyways.
Updated by okurz almost 4 years ago
- Copied to action #92788: Use openQA archiving feature on osd size:S added