Project

General

Profile

action #18006

continuous testing + delivery of tested openQA on openSUSE

Added by okurz over 5 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Organisational
Target version:
-
Start date:
2017-03-24
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

user story

As openSUSE fanboys we want to have openQA in oS:Fctry whenever it was tested on an openSUSE system so that we convince others and can enable safer continuous deployment

acceptance criteria

  • AC1: openQA packages with dependencies from obs://devel:openQA are submitted to oS:Fctry automatically after the openQA-in-openQA test passes

tasks

  • create an automated way to
    • copy package/project to test from devel:openQA to devel:openQA:testing
    • trigger openQA-in-openQA
    • after openQA-in-openQA monitor job succeeds, trigger a submission with osc sr from that project
  • if that works we could also submit new stable versions to released versions as maintenance requests. Because this is only accepted using bug reports we could auto-create the bug reports as well :)

Related issues

Related to openQA Project - action #26890: Warn users about version incompatibilityRejected2017-10-19

Related to openQA Project - action #16374: openQA stable versions with manual changelog/release notesRejected2017-01-31

Related to openQA Project - action #12972: extend self test for SLE: openQA fails to install on SLES12SP1: http://lord.arch/tests/2427#step/openqa_webui/11 nothing provides git-coreRejected2016-08-01

Related to openQA Infrastructure - action #55301: semi-automatic deployments on osdResolved2019-08-09

Related to openQA Project - action #55304: continuous deployment on o3Resolved2019-08-09

History

#1 Updated by coolo over 5 years ago

Note that factory does not accept _services - so would you need to register what git version actually was tested and submit that explicitly by regenerating the tar

#2 Updated by okurz over 5 years ago

so checkout package locally, repack, submit to devel:openQA:testing and submit from there?

#3 Updated by coolo over 5 years ago

yes, if you want to submit it, you have to fork off before testing

#4 Updated by okurz over 5 years ago

  • Status changed from In Progress to Feedback

created new subproject https://build.opensuse.org/project/show/devel:openQA:tested (as proposed in todays jangouts call, people like "tested" more than "testing"). update script in https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1

created two packages "os-autoinst" and "openQA" by "osc copypac" and adjustments:
https://build.opensuse.org/package/show/devel:openQA:tested/os-autoinst
https://build.opensuse.org/package/show/devel:openQA:tested/openQA

and created two submissions
https://build.opensuse.org/request/show/485999
https://build.opensuse.org/request/show/486014

they are declined already. Needs security audit for dbus service file https://bugzilla.suse.com/show_bug.cgi?id=1032649

so waiting for that

#5 Updated by okurz over 5 years ago

https://bugzilla.suse.com/show_bug.cgi?id=1032649 is RESOLVED FIXED but was until very recently waiting for https://build.opensuse.org/request/show/491831 to be resolved to have updated RPM check scripts. The SR got accepted 4 days ago, potentially it needs a bit of time until it's active on the side of Factory submission checks.

In the meantime I patched the spec files within :tested to not install the dbus services part to give legal checking a chance. So two new SRs exist

after these got accepted I can put the automatic submissions based on test results in place to update the existing packages.

#6 Updated by okurz about 5 years ago

both SR accepted, os-autoinst+openQA are in openSUSE Factory now \o/

#7 Updated by okurz about 5 years ago

next step: re-enable dbus part in spec file, then automatic submissions

#8 Updated by okurz about 5 years ago

https://build.opensuse.org/request/show/491831 was accepted 6 days ago. It does not seem to have an effect so far -> https://build.opensuse.org/request/show/494326 . any way to check version of rpmlint installed or something? I could only find rpmlint-mini and rpmlint-Factory in the build log
[10 May 2017 17:08:47] <|Anna|> sr#491831 [submit devel:openSUSE:Factory:rpmlint/rpmlint -> openSUSE:Factory/rpmlint] - whitelisting backintime (https://bugzilla.suse.com/1007723, https://bugzilla.suse.com/1032717) (forwarded request 49... (State: accepted) -- https://build.opensuse.org/request/show/491831
[10 May 2017 17:08:48] <|Anna|> sr#494326 [submit devel:openQA:tested/os-autoinst -> openSUSE:Factory/os-autoinst] Update including re-enabled dbus service installation (State: declined) -- https://build.opensuse.org/request/show/494326

#9 Updated by okurz about 5 years ago

rpmlint is in Factory now, os-autoinst builds and passes rpmlint check with enabled dbus. maxlin built with nogroup fix however there is another new dbus packaged https://build.opensuse.org/package/show/home:mlin7442:branches:devel:openQA:tested/openQA
I created a request for security audit: https://bugzilla.suse.com/show_bug.cgi?id=1039290

os-autoinst is in openSUSE:Factory but I would like to wait for the updated package of openQA before adding automatic submissions to prevent os-autoinst+openQA being out-of-sync and having incompatible versions.

#10 Updated by okurz about 5 years ago

waiting for https://build.opensuse.org/request/show/500158 reg. security audit of openQA dbus service

#11 Updated by okurz about 5 years ago

accepted with https://build.opensuse.org/request/show/500576, bug 1039290 VERIFIED FIXED, openQA with enabled dbus service is now in openSUSE Factory -> https://build.opensuse.org/request/show/503557

https://build.opensuse.org/request/show/504232 should bring devel:openQA/openQA in line with devel:openQA:tested/openQA, next step can be automatic submissions to openSUSE Factory then.

In the meantime, submitting to openSUSE Leap 42.3 needs sr#504635 and sr#504637 first for updated dependencies

#12 Updated by okurz about 5 years ago

https://build.opensuse.org/request/show/504915 to bring devel:openQA/openQA in sync with devel:openQA:tested/openQA accepted. Also accepted for Leap 42.3:

so os-autoinst and openQA is in openSUSE Leap 42.3 \o/

so now for the automatic submissions :-)

#13 Updated by okurz almost 5 years ago

  • Description updated (diff)
  • Status changed from Feedback to In Progress

Updated the "update" package within devel:openQA:tested to submit new versions of os-autoinst as well as openQA and also (unverified) overwriting previous submit requests if any:
https://build.opensuse.org/package/rdiff/devel:openQA:tested/update?linkrev=base&rev=5

Adapted http://lord.arch.suse.de:8080/job/submit-openQA-TW-to-oS_Fctry/ to call

$(osc cat devel:openQA:tested update _update.sh)

to run the script after the openQA-in-openQA tests work fine.

Let's see if my credentials are properly picked up.
See http://lord.arch.suse.de:8080/job/monitor-openQA_in_openQA-TW/ and its downstream job.

#14 Updated by okurz almost 5 years ago

  • Status changed from In Progress to Feedback

http://lord.arch.suse.de:8080/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/187/ did a successful run with automatic submit requests created for os-autoinst and openQA using https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1

That should trigger now whenever os-autoinst and openQA trigger successful tests of openQA-in-openQA.

The acceptance criterion is covered by this but I want to collect feedback based on the content of the submit requests and how they are handled by the human reviewers :)

#15 Updated by okurz almost 5 years ago

  • Status changed from Feedback to In Progress

suggestion by coolo: "stop creating https://build.opensuse.org/request/show/539241. don't run the service again - but osc up -S devel:openQA/os-autoinst and mv _service:* to the files in devel:openQA:tested/os-autoinst"

disabled http://lord.arch.suse.de:8080/job/trigger-openQA_in_openQA-TW/, should try the different update workflow and then enable again.

#16 Updated by okurz over 4 years ago

  • Related to action #26890: Warn users about version incompatibility added

#17 Updated by okurz over 4 years ago

  • Status changed from In Progress to Feedback

so coolo did some changes to https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1 to do the update properly without rebuilding everything and not overwriting the changelog.

http://lord.arch.suse.de:8080/job/submit-openQA-TW-to-oS_Fctry/195/console shows the log output of the successful execution of the submission. https://build.opensuse.org/request/show/556688 is the generated SR for os-autoinst.

So let's see if somebody complains :)

#18 Updated by okurz over 4 years ago

So far there have been some automatic SRs, nobody complained. Submissions to openSUSE Leap 15.0 also happen automatically after the initial manual one. Now as our main production workers run stable version of openSUSE Leap the maintenance updates would need to be done automatically as well. astieger would be ok with say one submission per week but I'm not sure if this is responsive enough. Maybe just trigger the automatic updates on o3 and osd from devel:openQA:tested whenever the openQA-in-openQA tests succeed?

#19 Updated by coolo over 4 years ago

We won't deploy updates automatically, no

#21 Updated by okurz over 3 years ago

coolo wrote:

We won't deploy updates automatically, no

Well, that problem had been solved by rbrownsuse at least for o3 workers with changing the setup to "transactional-update" systems ;)

#22 Updated by okurz over 3 years ago

What do you think of "one-click-deployments" like in https://gitlab.suse.de/okurz/osd-deployment/-/jobs/54944 including automatic email notification of deployed changes?

#23 Updated by okurz over 3 years ago

What changed since #18006#note-19

  • o3 workers are based on openSUSE Leap 15.0 transactional server setup by rbrown with automatic nightly deployment from devel:openQA
  • o3 webui is not automatically updated but regularly about once a week by me

Proposal

  • move the existing jenkins jobs into a gitlab CI pipeline (see #18006#note-22)
  • as in before create the SR to openSUSE Factory but in parallel submit from devel:openQA:tested to devel:openQA:stable
  • configure our workers to install from devel:openQA:stable instead of devel:openQA
  • optional: Extend the automatic tests to test on openSUSE Leap 42.3 base as well, not only Tumbleweed
  • configure the webui host to also install from devel:openQA:stable and do that as part of our gitlab CI pipeline automatically

#24 Updated by okurz over 3 years ago

  • Related to action #16374: openQA stable versions with manual changelog/release notes added

#25 Updated by okurz about 3 years ago

discussed with nsinger, we can try out the gitlab pipeline approach this Wednesday for OSD.

side-note: I wonder why we store https://build.opensuse.org/package/show/devel:openQA:tested/update on OBS and not a git repo, could be more accessible for others e.g. for other packages to use as a template.

#26 Updated by okurz about 3 years ago

  • Related to action #12972: extend self test for SLE: openQA fails to install on SLES12SP1: http://lord.arch/tests/2427#step/openqa_webui/11 nothing provides git-core added

#27 Updated by okurz about 3 years ago

I do not recall any good reason to try to manage the single script _update.sh in OBS when we have even moved the spec files to the upstream github projects hence in #54467 I proposed to create https://github.com/os-autoinst/scripts which I finally did. So therefore I proposed https://github.com/os-autoinst/scripts/pull/3 with the monitor+trigger scripts added along with the automatic submission script in https://github.com/os-autoinst/scripts/pull/2 to bring these components together. This could even allow us to use external CIs like the CI on gitlab.com or travis CI or circle CI or whatever instead of relying on an internal jenkins instance.

#28 Updated by okurz about 3 years ago

both PRs merged. https://gitlab.suse.de/openqa/scripts/merge_requests/376 created to propose the deletion of the openqa-in-openqa scripts from https://gitlab.suse.de/openqa/scripts/ as they now live in https://github.com/os-autoinst/scripts . Updated the description on https://build.opensuse.org/package/show/devel:openQA:tested/update to mark it as OBSOLETE with a link to https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit instead but have not yet deleted it.

For automatic installs of the openQA webUI I am thinking of checking:

result="$(curl -s "https://api.opensuse.org/public/build/devel:openQA/_result?repository=openSUSE_Leap_15.1&arch=x86_64")"
echo -e "$result" | grep -q 'project.*state="published"'
echo "$result" | grep 'status package' | (! grep -v '\(disabled\|succeeded\|excluded\)')

during a nightly window. This should also cover neighboring packages, e.g. openQA-test. TODO how to handle that in a cron job, e.g. wait for X retries until package is published and fine?

I wonder why we proposed devel:openQA:tested and not devel:openQA:testing when the package are not yet tested when they enter that project? I am thinking about a subproject to put packages to after they actually got tested
Would it be a good idea to have a subproject with actually tested packages to auto-install from? or should I just wait for AMQP events from OBS telling me that all packages are successfully published to trigger an install? or a gitlab CI pipeline?

#29 Updated by okurz almost 3 years ago

Let's try the easy/ugly way. on o3:

cat /etc/cron.d/auto-update
# https://progress.opensuse.org/issues/18006
0 3 * * 0 root result="$(curl -s "https://api.opensuse.org/public/build/devel:openQA/_result?repository=openSUSE_Leap_15.1&arch=x86_64")" && echo -e "$result" | grep -q 'project.*state="published"' && echo "$result" | grep 'status package' | (! grep -v '\(disabled\|succeeded\|excluded\)') && zypper -n dup --replacefiles --auto-agree-with-licenses --force-resolution --download-in-advance

#30 Updated by cdywan almost 3 years ago

how to handle that in a cron job, e.g. wait for X retries until package is published and fine?

What happens if the cron jobsystemd timer, which is hosted in git, fails after X attempts? Do we get an email/ notification?

Some thoughts on the one-click deployment, after having been the substitute deployment lead:

  • The manual steps involve these assets, which we probably want to retain
    • A backup
    • Versions from before and after
    • Install logs - We'd obviously get logs from the whole process, but it might be nice to save it explicitly for convenience
    • Generating a changelog
    • Updating the deployment page
    • Sending an email with the deployment
  • Would we want to have separate worker and webui jobs? Case in point, poo#11706 benefits from deploying workers before webui

#31 Updated by okurz almost 3 years ago

  • Related to action #55301: semi-automatic deployments on osd added

#32 Updated by okurz almost 3 years ago

#33 Updated by okurz almost 3 years ago

  • Status changed from Feedback to Resolved

cdywan wrote:

how to handle that in a cron job, e.g. wait for X retries until package is published and fine?

What happens if the cron jobsystemd timer, which is hosted in git, fails after X attempts? Do we get an email/ notification?

I don't know to what timer you refer to. So far I am not aware of a timer hosted in git so we are discussing something hypothetical: In this case I would retry multiple times but abort after X attempts with an email to the admins. There is a mailing list o3-admins@suse.de for this. However currently I am trying even with no retries in the cron job. I assume that this actually works in most cases assuming that the repo would be pretty often just stable during the European night :)

Some thoughts on the one-click deployment, after having been the substitute deployment lead:

  • The manual steps involve these assets, which we probably want to retain
    • A backup
    • Versions from before and after
    • Install logs - We'd obviously get logs from the whole process, but it might be nice to save it explicitly for convenience
    • Generating a changelog
    • Updating the deployment page
    • Sending an email with the deployment

well, the focus for this ticket – as the subject says – should be "continuous testing + delivery of tested openQA on openSUSE" which to be honest we have covered since a year :) I am kinda "abusing" the ticket to bring forward my ideas of "continuous deployment" as in, whenever there is any change of either os-autoinst or openQA we should deploy, as often as possible. This for me means that the version from before and after become easier to track (smaller diff) but makes "updating the deployment page" as well as "sending an email with the deployment" less useful or even spammy, annoying and in the end unnecessary.

  • Would we want to have separate worker and webui jobs? Case in point, poo#11706 benefits from deploying workers before webui

no, we want – or "I prefer very much" :) – changes which are backward- and forward compatible. Having a more simple deploy pipeline which would even deploy every single update of either os-autoinst or openQA would motivate more to do this properly in code :)

I think it's time to close this ticket and let's have two separate follow-up versions, one for "continuous deployment on o3" #55304 and another one for "semi-automatic deployments on osd" #55301 :)

Also available in: Atom PDF