Project

General

Profile

action #94838

QA - coordination #99303: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, approval and release

Make qem-dashboard a proper public open source project size:M

Added by okurz over 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2021-06-29
Due date:
2021-12-18
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

I wonder why I even need to write this, we are working at SUSE, right? :) (If still not convinced see opensource.suse.com/ and https://intra.suse.net/company/living-our-values-everyday/ and if you don't want to talk to me about this then talk to others, e.g. bzoltan1)

Acceptance criteria

Suggestions

  • Consider openqa_review as an existing example using both git clone in a gitlab pipeline and packages built
  • git fetch from gitlab

Related issues

Related to QA - action #97274: qam dashboard improvement ideasNew2021-06-29

Blocks QA - action #94588: [qem] dashboard.qam.suse.de to be included in approval qam-openqaNew2021-06-23

Copied to openQA Project - action #95821: Make qa-maintenance/openQABot a proper public open source projectNew2021-06-29

Copied to openQA Project - action #102206: Make bot-ng a proper public open source project size:MResolved2021-06-29

History

#1 Updated by okurz over 1 year ago

  • Blocks action #94588: [qem] dashboard.qam.suse.de to be included in approval qam-openqa added

#2 Updated by kraih over 1 year ago

I also prefer making everything open source from the start. Main reasons not to do so here were that, a) the project was just a proof of concept until recently, b) openQABot and SMELT are closed source, c) ease of deployment from our gitlab instance, since Jan already has ansible set up there for pipelines. Aside from the code being pretty much useless without having a SMELT instance, the main concern will be deployment from GitHub to the dashboard machine. Oh, and of course naming the project. :)

#3 Updated by okurz over 1 year ago

  • Copied to action #95821: Make qa-maintenance/openQABot a proper public open source project added

#4 Updated by cdywan over 1 year ago

As I understand it there's nothing blocking us here besides finding a new name. If CI can't easily be setup on GitHub I wouldn't consider that a problem, since this is about development and not about production deployments.

#5 Updated by kraih over 1 year ago

cdywan wrote:

As I understand it there's nothing blocking us here besides finding a new name. If CI can't easily be setup on GitHub I wouldn't consider that a problem, since this is about development and not about production deployments.

Deployment to production is currently triggered by commits to the repo (after all tests pass in CI).

#6 Updated by okurz over 1 year ago

  • Related to action #97274: qam dashboard improvement ideas added

#7 Updated by okurz about 1 year ago

  • Description updated (diff)
  • Target version changed from future to Ready

#8 Updated by okurz about 1 year ago

  • Copied to action #102206: Make bot-ng a proper public open source project size:M added

#9 Updated by cdywan about 1 year ago

  • Subject changed from Make qem-dashboard a proper public open source project to Make qem-dashboard a proper public open source project size:M
  • Description updated (diff)
  • Status changed from New to Workable

#10 Updated by kraih 12 months ago

  • Assignee set to kraih

Asked Jan for ideas how to deploy from GitHub.

#11 Updated by jbaier_cz 12 months ago

cdywan wrote:

If CI can't easily be setup on GitHub I wouldn't consider that a problem, since this is about development and not about production deployments.

Well, I would consider not being able to deploy a major usability issue. What is the development for if you cannot use it?

Suggestions:

Anyways, in the current state the deployment is orchestrated via Ansible playbook (which actually just prepares some service files and does a git checkout). This is managed in a separate gitlab project qa-maintenance/qamops and will work as long as we will have read access to the qem-dashboard code repository (which should be OK for a public GitHub repository). The only think here would be to trigger the deployment pipeline itself. Currently, we are using multi-project pipelines and we are triggering the downstream pipeline via CI inside the qem-dashboard pipeline. I see several options here, each with its own ups and downs (ordered by simplicity of the solution):

  1. Triggering pipeline through API directly from GitHub
    con: we need gitlab.suse.de to be accessible from GitHub for the webhook to work

  2. Pull mirroring from the remote and triggering pipeline for updates
    con: pull mirroring is a premium feature in GitLab

  3. Running the deployment in a scheduled pipeline
    con: not event-based, the deployed version will not be always up-to-date

  4. Push mirroring to GitHub
    con: the development will still be done in GitLab

  5. Add OBS/IBS to the mix and use workflow integration to build a package after GitHub push and trigger package installation after successful build
    con: complicated setup

#12 Updated by okurz 12 months ago

jbaier_cz wrote:

  1. Running the deployment in a scheduled pipeline con: not event-based, the deployed version will not be always up-to-date

I recommend to go with this and run it every couple of minutes. I see no harm in doing that every like 10 minutes with an early-abort if the current git version matches the last deployed one.

#13 Updated by jbaier_cz 12 months ago

The con here is a lot of wasted runs (with tight schedules, like every 10 minutes). The early abort is not very feasible without rewriting the current playbook as the git operation is at the very end (and effectively cannot be done before you make sure the project directory exists and git binary is installed; so we are talking here only about skipping systemd unit files creation). Running the playbook often has also some other drawbacks.

#14 Updated by kraih 12 months ago

  • Status changed from Workable to In Progress

#15 Updated by kraih 12 months ago

jbaier_cz wrote:

The con here is a lot of wasted runs (with tight schedules, like every 10 minutes).

As long as the pipeline can also be triggered manually, even just running it every 24 hours with the schedule would be fine.

#16 Updated by jbaier_cz 12 months ago

kraih wrote:

jbaier_cz wrote:

The con here is a lot of wasted runs (with tight schedules, like every 10 minutes).

As long as the pipeline can also be triggered manually, even just running it every 24 hours with the schedule would be fine.

Yes, once a day and manual option to run the pipeline seems fine to me as well.

#17 Updated by kraih 12 months ago

The public repo exists now. Next step will be the deployment pipeline. It's currently deployed from a manual checkout. https://github.com/openSUSE/qem-dashboard

#18 Updated by jbaier_cz 12 months ago

Deployment book updated with https://gitlab.suse.de/qa-maintenance/qamops/-/commit/c90aa3e25ed8ca04a2d89f369b23d843804745fa

Now we just need to update the original pipeline add the manual trigger and scheduled pipeline for deploy.

#19 Updated by jbaier_cz 12 months ago

Looking at the original pipeline... did someone mention the tests?

#20 Updated by openqa_review 12 months ago

  • Due date set to 2021-12-18

Setting due date based on mean cycle time of SUSE QE Tools

#21 Updated by kraih 12 months ago

jbaier_cz wrote:

Looking at the original pipeline... did someone mention the tests?

The tests now run on GitHub, no need to bother with them in GitLab anymore. All we need is deployment.

#22 Updated by jbaier_cz 12 months ago

I added the new deployment steps and created update for the pipeline to skip runs on commit and allow deploys on demand via schedule.

#23 Updated by kraih 12 months ago

  • Status changed from In Progress to Feedback

After some dependency tweaks and help from Jan the deployment mostly works now. We'll just have to add dashboard-amqp-watcher to the services that are restarted after deployment.

#24 Updated by jbaier_cz 12 months ago

Added in commit bde85835

#25 Updated by kraih 12 months ago

  • Status changed from Feedback to Resolved

#26 Updated by cdywan 11 months ago

  • Parent task set to #80194

Also available in: Atom PDF