Project

General

Profile

Actions
Copy link

action #105379

closed

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #80908: [epic] Continuous deployment (package upgrade or config update) without interrupting currently running openQA jobs

Continuous deployment of o3 workers - one worker first size:M

Added by okurz over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
mkittler
Category:
Feature requests
Target version:
Ready
Start date:
2022-01-24
Due date:
% Done:

0%

Estimated time:

Description

Acceptance criteria

  • AC1: o3 workers automatically deploy after every update to the package os-autoinst
  • AC2: No significant downtime due to updates

Suggestions

  • Just do this on one machine, extend to others in a later ticket* We can likely change our transactional workers to have a read-writable root partition while still doing a nightly transactional update and reboot. We effectively already do this on openqaworker7 which has a r/w root. So likely just a change in /etc/fstab while keeping the services transactional-update and rebootmgr in place. If you don't want to do that then just use one of the machines with r/w root
  • Try out zypper -n ref -r devel:openQA | grep -q 'is up to date' && zypper -n dup -r devel:openQA ||: in a systemd timer every 5 minutes
  • Include the change in github.com/os-autoinst/openQA/, updating or relating to https://github.com/os-autoinst/openQA/blob/master/systemd/opensuse/openqa-auto-update.service
  • Write down the exact commands being used so that we can extend the approach to other machines
  • Optional: Include the package openQA-worker
  • Optional: First try on one of our o3 workers, then extend to others

Related issues 4 (0 open — 4 closed)

Related to openQA Project - action #109851: os-autoinst was removed from o3 openqaworker7Resolvedmkittler2022-04-12

Actions
Related to openQA Infrastructure - action #111758: o3 jobs exceeding MAX_SETUP_TIME auto_review:"(?s)openqaworker4.*timeout: setup exceeded MAX_SETUP_TIME":retry size:MResolvedfavogt2022-05-30

Actions
Copied to openQA Project - action #105885: Continuous deployment of o3 workers - all the other o3 workers size:MResolvedmkittler

Actions
Copied to openQA Project - action #111989: Seems like o3 machines do not automatically reboot anymore, likely because we continuously call `zypper dup` so that the nightly upgrades don't find any changes? size:MResolvedokurz2022-01-24

Actions
Actions
Copy link

Also available in: Atom PDF