action #18006
closedcontinuous testing + delivery of tested openQA on openSUSE
0%
Description
user story¶
As openSUSE fanboys we want to have openQA in oS:Fctry whenever it was tested on an openSUSE system so that we convince others and can enable safer continuous deployment
acceptance criteria¶
- AC1: openQA packages with dependencies from obs://devel:openQA are submitted to oS:Fctry automatically after the openQA-in-openQA test passes
tasks¶
- create an automated way to
- copy package/project to test from devel:openQA to devel:openQA:testing
- trigger openQA-in-openQA
- after openQA-in-openQA monitor job succeeds, trigger a submission with
osc sr
from that project
- if that works we could also submit new stable versions to released versions as maintenance requests. Because this is only accepted using bug reports we could auto-create the bug reports as well :)
Updated by coolo over 7 years ago
Note that factory does not accept _services - so would you need to register what git version actually was tested and submit that explicitly by regenerating the tar
Updated by okurz over 7 years ago
so checkout package locally, repack, submit to devel:openQA:testing and submit from there?
Updated by coolo over 7 years ago
yes, if you want to submit it, you have to fork off before testing
Updated by okurz over 7 years ago
- Status changed from In Progress to Feedback
created new subproject https://build.opensuse.org/project/show/devel:openQA:tested (as proposed in todays jangouts call, people like "tested" more than "testing"). update script in https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1
created two packages "os-autoinst" and "openQA" by "osc copypac" and adjustments:
https://build.opensuse.org/package/show/devel:openQA:tested/os-autoinst
https://build.opensuse.org/package/show/devel:openQA:tested/openQA
and created two submissions
https://build.opensuse.org/request/show/485999
https://build.opensuse.org/request/show/486014
they are declined already. Needs security audit for dbus service file https://bugzilla.suse.com/show_bug.cgi?id=1032649
so waiting for that
Updated by okurz over 7 years ago
https://bugzilla.suse.com/show_bug.cgi?id=1032649 is RESOLVED FIXED but was until very recently waiting for https://build.opensuse.org/request/show/491831 to be resolved to have updated RPM check scripts. The SR got accepted 4 days ago, potentially it needs a bit of time until it's active on the side of Factory submission checks.
In the meantime I patched the spec files within :tested to not install the dbus services part to give legal checking a chance. So two new SRs exist
- os-autoinst already accepted
- openQA in review
after these got accepted I can put the automatic submissions based on test results in place to update the existing packages.
Updated by okurz over 7 years ago
both SR accepted, os-autoinst+openQA are in openSUSE Factory now \o/
Updated by okurz over 7 years ago
next step: re-enable dbus part in spec file, then automatic submissions
Updated by okurz over 7 years ago
https://build.opensuse.org/request/show/491831 was accepted 6 days ago. It does not seem to have an effect so far -> https://build.opensuse.org/request/show/494326 . any way to check version of rpmlint installed or something? I could only find rpmlint-mini and rpmlint-Factory in the build log
[10 May 2017 17:08:47] <|Anna|> sr#491831 [submit devel:openSUSE:Factory:rpmlint/rpmlint -> openSUSE:Factory/rpmlint] - whitelisting backintime (https://bugzilla.suse.com/1007723, https://bugzilla.suse.com/1032717) (forwarded request 49... (State: accepted) -- https://build.opensuse.org/request/show/491831
[10 May 2017 17:08:48] <|Anna|> sr#494326 [submit devel:openQA:tested/os-autoinst -> openSUSE:Factory/os-autoinst] Update including re-enabled dbus service installation (State: declined) -- https://build.opensuse.org/request/show/494326
Updated by okurz over 7 years ago
rpmlint is in Factory now, os-autoinst builds and passes rpmlint check with enabled dbus. maxlin built with nogroup fix however there is another new dbus packaged https://build.opensuse.org/package/show/home:mlin7442:branches:devel:openQA:tested/openQA
I created a request for security audit: https://bugzilla.suse.com/show_bug.cgi?id=1039290
os-autoinst is in openSUSE:Factory but I would like to wait for the updated package of openQA before adding automatic submissions to prevent os-autoinst+openQA being out-of-sync and having incompatible versions.
Updated by okurz over 7 years ago
waiting for https://build.opensuse.org/request/show/500158 reg. security audit of openQA dbus service
Updated by okurz over 7 years ago
accepted with https://build.opensuse.org/request/show/500576, bug 1039290 VERIFIED FIXED, openQA with enabled dbus service is now in openSUSE Factory -> https://build.opensuse.org/request/show/503557
https://build.opensuse.org/request/show/504232 should bring devel:openQA/openQA in line with devel:openQA:tested/openQA, next step can be automatic submissions to openSUSE Factory then.
In the meantime, submitting to openSUSE Leap 42.3 needs sr#504635 and sr#504637 first for updated dependencies
Updated by okurz over 7 years ago
https://build.opensuse.org/request/show/504915 to bring devel:openQA/openQA in sync with devel:openQA:tested/openQA accepted. Also accepted for Leap 42.3:
- https://build.opensuse.org/request/show/503610
- https://build.opensuse.org/request/show/504636
- https://build.opensuse.org/request/show/504637
so os-autoinst and openQA is in openSUSE Leap 42.3 \o/
so now for the automatic submissions :-)
Updated by okurz about 7 years ago
- Description updated (diff)
- Status changed from Feedback to In Progress
Updated the "update" package within devel:openQA:tested to submit new versions of os-autoinst as well as openQA and also (unverified) overwriting previous submit requests if any:
https://build.opensuse.org/package/rdiff/devel:openQA:tested/update?linkrev=base&rev=5
Adapted http://lord.arch.suse.de:8080/job/submit-openQA-TW-to-oS_Fctry/ to call
$(osc cat devel:openQA:tested update _update.sh)
to run the script after the openQA-in-openQA tests work fine.
Let's see if my credentials are properly picked up.
See http://lord.arch.suse.de:8080/job/monitor-openQA_in_openQA-TW/ and its downstream job.
Updated by okurz about 7 years ago
- Status changed from In Progress to Feedback
http://lord.arch.suse.de:8080/view/openQA-in-openQA/job/submit-openQA-TW-to-oS_Fctry/187/ did a successful run with automatic submit requests created for os-autoinst and openQA using https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1
That should trigger now whenever os-autoinst and openQA trigger successful tests of openQA-in-openQA.
The acceptance criterion is covered by this but I want to collect feedback based on the content of the submit requests and how they are handled by the human reviewers :)
Updated by okurz about 7 years ago
- Status changed from Feedback to In Progress
suggestion by coolo: "stop creating https://build.opensuse.org/request/show/539241. don't run the service again - but osc up -S devel:openQA/os-autoinst and mv _service:* to the files in devel:openQA:tested/os-autoinst"
disabled http://lord.arch.suse.de:8080/job/trigger-openQA_in_openQA-TW/, should try the different update workflow and then enable again.
Updated by okurz about 7 years ago
- Related to action #26890: Warn users about version incompatibility added
Updated by okurz about 7 years ago
- Status changed from In Progress to Feedback
so coolo did some changes to https://build.opensuse.org/package/view_file/devel:openQA:tested/update/_update.sh?expand=1 to do the update properly without rebuilding everything and not overwriting the changelog.
http://lord.arch.suse.de:8080/job/submit-openQA-TW-to-oS_Fctry/195/console shows the log output of the successful execution of the submission. https://build.opensuse.org/request/show/556688 is the generated SR for os-autoinst.
So let's see if somebody complains :)
Updated by okurz almost 7 years ago
So far there have been some automatic SRs, nobody complained. Submissions to openSUSE Leap 15.0 also happen automatically after the initial manual one. Now as our main production workers run stable version of openSUSE Leap the maintenance updates would need to be done automatically as well. astieger would be ok with say one submission per week but I'm not sure if this is responsive enough. Maybe just trigger the automatic updates on o3 and osd from devel:openQA:tested whenever the openQA-in-openQA tests succeed?
Updated by okurz about 6 years ago
coolo wrote:
We won't deploy updates automatically, no
Well, that problem had been solved by rbrownsuse at least for o3 workers with changing the setup to "transactional-update" systems ;)
Updated by okurz about 6 years ago
What do you think of "one-click-deployments" like in https://gitlab.suse.de/okurz/osd-deployment/-/jobs/54944 including automatic email notification of deployed changes?
Updated by okurz almost 6 years ago
What changed since #18006#note-19¶
- o3 workers are based on openSUSE Leap 15.0 transactional server setup by rbrown with automatic nightly deployment from devel:openQA
- o3 webui is not automatically updated but regularly about once a week by me
Proposal¶
- move the existing jenkins jobs into a gitlab CI pipeline (see #18006#note-22)
- as in before create the SR to openSUSE Factory but in parallel submit from devel:openQA:tested to devel:openQA:stable
- configure our workers to install from devel:openQA:stable instead of devel:openQA
- optional: Extend the automatic tests to test on openSUSE Leap 42.3 base as well, not only Tumbleweed
- configure the webui host to also install from devel:openQA:stable and do that as part of our gitlab CI pipeline automatically
Updated by okurz almost 6 years ago
- Related to action #16374: openQA stable versions with manual changelog/release notes added
Updated by okurz over 5 years ago
discussed with nsinger, we can try out the gitlab pipeline approach this Wednesday for OSD.
side-note: I wonder why we store https://build.opensuse.org/package/show/devel:openQA:tested/update on OBS and not a git repo, could be more accessible for others e.g. for other packages to use as a template.
Updated by okurz over 5 years ago
- Related to action #12972: extend self test for SLE: openQA fails to install on SLES12SP1: http://lord.arch/tests/2427#step/openqa_webui/11 nothing provides git-core added
Updated by okurz over 5 years ago
I do not recall any good reason to try to manage the single script _update.sh
in OBS when we have even moved the spec files to the upstream github projects hence in #54467 I proposed to create https://github.com/os-autoinst/scripts which I finally did. So therefore I proposed https://github.com/os-autoinst/scripts/pull/3 with the monitor+trigger scripts added along with the automatic submission script in https://github.com/os-autoinst/scripts/pull/2 to bring these components together. This could even allow us to use external CIs like the CI on gitlab.com or travis CI or circle CI or whatever instead of relying on an internal jenkins instance.
Updated by okurz over 5 years ago
both PRs merged. https://gitlab.suse.de/openqa/scripts/merge_requests/376 created to propose the deletion of the openqa-in-openqa scripts from https://gitlab.suse.de/openqa/scripts/ as they now live in https://github.com/os-autoinst/scripts . Updated the description on https://build.opensuse.org/package/show/devel:openQA:tested/update to mark it as OBSOLETE with a link to https://github.com/os-autoinst/scripts/blob/master/os-autoinst-obs-auto-submit instead but have not yet deleted it.
For automatic installs of the openQA webUI I am thinking of checking:
result="$(curl -s "https://api.opensuse.org/public/build/devel:openQA/_result?repository=openSUSE_Leap_15.1&arch=x86_64")"
echo -e "$result" | grep -q 'project.*state="published"'
echo "$result" | grep 'status package' | (! grep -v '\(disabled\|succeeded\|excluded\)')
during a nightly window. This should also cover neighboring packages, e.g. openQA-test. TODO how to handle that in a cron job, e.g. wait for X retries until package is published and fine?
I wonder why we proposed devel:openQA:tested and not devel:openQA:testing when the package are not yet tested when they enter that project? I am thinking about a subproject to put packages to after they actually got tested
Would it be a good idea to have a subproject with actually tested packages to auto-install from? or should I just wait for AMQP events from OBS telling me that all packages are successfully published to trigger an install? or a gitlab CI pipeline?
Updated by okurz over 5 years ago
Let's try the easy/ugly way. on o3:
cat /etc/cron.d/auto-update
# https://progress.opensuse.org/issues/18006
0 3 * * 0 root result="$(curl -s "https://api.opensuse.org/public/build/devel:openQA/_result?repository=openSUSE_Leap_15.1&arch=x86_64")" && echo -e "$result" | grep -q 'project.*state="published"' && echo "$result" | grep 'status package' | (! grep -v '\(disabled\|succeeded\|excluded\)') && zypper -n dup --replacefiles --auto-agree-with-licenses --force-resolution --download-in-advance
Updated by livdywan over 5 years ago
how to handle that in a cron job, e.g. wait for X retries until package is published and fine?
What happens if the cron jobsystemd timer, which is hosted in git, fails after X attempts? Do we get an email/ notification?
Some thoughts on the one-click deployment, after having been the substitute deployment lead:
- The manual steps involve these assets, which we probably want to retain
- A backup
- Versions from before and after
- Install logs - We'd obviously get logs from the whole process, but it might be nice to save it explicitly for convenience
- Generating a changelog
- Updating the deployment page
- Sending an email with the deployment
- Would we want to have separate worker and webui jobs? Case in point, poo#11706 benefits from deploying workers before webui
Updated by okurz over 5 years ago
- Related to action #55301: semi-automatic deployments on osd added
Updated by okurz over 5 years ago
- Related to action #55304: continuous deployment on o3 added
Updated by okurz over 5 years ago
- Status changed from Feedback to Resolved
cdywan wrote:
how to handle that in a cron job, e.g. wait for X retries until package is published and fine?
What happens if the cron jobsystemd timer, which is hosted in git, fails after X attempts? Do we get an email/ notification?
I don't know to what timer you refer to. So far I am not aware of a timer hosted in git so we are discussing something hypothetical: In this case I would retry multiple times but abort after X attempts with an email to the admins. There is a mailing list o3-admins@suse.de for this. However currently I am trying even with no retries in the cron job. I assume that this actually works in most cases assuming that the repo would be pretty often just stable during the European night :)
Some thoughts on the one-click deployment, after having been the substitute deployment lead:
- The manual steps involve these assets, which we probably want to retain
- A backup
- Versions from before and after
- Install logs - We'd obviously get logs from the whole process, but it might be nice to save it explicitly for convenience
- Generating a changelog
- Updating the deployment page
- Sending an email with the deployment
well, the focus for this ticket – as the subject says – should be "continuous testing + delivery of tested openQA on openSUSE" which to be honest we have covered since a year :) I am kinda "abusing" the ticket to bring forward my ideas of "continuous deployment" as in, whenever there is any change of either os-autoinst or openQA we should deploy, as often as possible. This for me means that the version from before and after become easier to track (smaller diff) but makes "updating the deployment page" as well as "sending an email with the deployment" less useful or even spammy, annoying and in the end unnecessary.
- Would we want to have separate worker and webui jobs? Case in point, poo#11706 benefits from deploying workers before webui
no, we want – or "I prefer very much" :) – changes which are backward- and forward compatible. Having a more simple deploy pipeline which would even deploy every single update of either os-autoinst or openQA would motivate more to do this properly in code :)
I think it's time to close this ticket and let's have two separate follow-up versions, one for "continuous deployment on o3" #55304 and another one for "semi-automatic deployments on osd" #55301 :)