action #12956
closedAdd tests for Krypton/Argon live cds
0%
Description
acceptance criteria¶
- AC1: Every new Krypton x86_64 build is tested on o3
tasks¶
- DONE: get Krypton to run locally -> http://lord.arch/tests/2376
- DONE: publish updated tests/needles
- DONE: create new job group(s) -> https://openqa.opensuse.org/group_overview/23
- DONE: implement trigger solution based on every change in the file in the download repo, i.e. if the name of the ISO changes -> http://lord.arch:8080/job/krypton-openqa/
- DONE: optional: use for marketing (see #12956#note-1) and #12956#note-9
- DONE: optional: test result feedback to OBS (see #12956#note-1 and #12956#note-2)
open questions¶
- should there be just one job group for all Krypton/Argon or two, one for Krypton, one for Argon, one even more because there are different variants of these, e.g. "Krypton stable"?
-> Krypton and Krypton stable shouldn't differ except for needles which might apply for Krypton before they will match Krypton stable during a later test run (when the stable build is updated to a former unstable)
- so how should we structure the Krypton/Krypton stable/Argon tests
- I propose ENV-FLAVOR-Krypton, ENV-FLAVOR-Krypton_stable, ENV-FLAVOR-Argon
- do we need a variable group all "noble gases"? e.g. "ENV-KDE_UNSTABLE-1"
further details¶
All images that we want to test are available under http://download.opensuse.org/repositories/KDE:/Medias/images/iso/
- what are the differences between openSUSE_Krypton_stable.x86_64-5.7.0-Build11.24.iso and openSUSE_Krypton.x86_64-5.6.90-Build4.51.iso on http://download.opensuse.org/repositories/KDE:/Medias/images/iso/? -> Krypton is latest git, Krypton stable latest release (beta, rc). The version number has to be hardcoded in the kiwi file any nobody updated that yet. fvogt updated kiwi file in this case
- how should openQA tests be triggered? -> okurz: I assume every change in the file in the download repo, i.e. if the name of the ISO changes -> yes
- can we feed back openQA results to OBS or the ones responsible for building Krypton/Argon? -> would be great, see #12956#note-1 and #12956#note-2
- can we promote this better, e.g. blog about it? -> see #12956#note-1
Always current test on o3 available as: https://openqa.opensuse.org/tests/latest?distri=opensuse&arch=x86_64&flavor=Krypton-Live&test=krypton-live
Updated by luca_b over 8 years ago
what are the differences between openSUSE_Krypton_stable.x86_64-5.7.0-Build11.24.iso and openSUSE_Krypton.x86_64-5.6.90-Build4.51.is
Lower version is probably just a hindsight. The ISOs not marked "stable" target the unstable branches and usually have the beta/RC release numbers. They need to be bumped after a new major release.
can we feed back openQA results to OBS or the ones responsible for building Krypton/Argon?
Yes, yes, yes! Those responsible would be the openSUSE community KDE team. This would be very useful as upstream bugs would be found and hopefully fixed. ;)
can we promote this better, e.g. blog about it?
So far I handled the communication but on my personal blog: see
- https://www.dennogumi.org/2016/06/two-in-one/
- https://www.dennogumi.org/2016/02/adding-wayland-to-the-gas-mixture/
- https://www.dennogumi.org/2016/02/where-are-my-noble-gases-i-need-more-noble-gases/
Not enough time to do more for now.
I'm not knowledgeable enough on openQA to answer the other questions.
Updated by favogt over 8 years ago
- should there be just one job group for all Krypton/Argon or two, one for Krypton, one for Argon, one even more because there are different variants of these, e.g. "Krypton stable"?
As Argon is based on Leap 42.1, it may behave differently. Krypton and Krypton stable shouldn't differ too much except for needles.
- what are the differences between openSUSE_Krypton_stable.x86_64-5.7.0-Build11.24.iso and openSUSE_Krypton.x86_64-5.6.90-Build4.51.iso on http://download.opensuse.org/repositories/KDE:/Medias/images/iso/?
Version number is changed now, but a rebuild will take a while due to the Akonadi splits.
- how should openQA tests be triggered? -> okurz: I assume every change in the file in the download repo, i.e. if the name of the ISO changes
Yup.
- can we feed back openQA results to OBS or the ones responsible for building Krypton/Argon?
Yes, but only if there is a way for the KDE team to update needles. Otherwise it may become annoying to have a ton of expected failures.
Updated by okurz over 8 years ago
- Description updated (diff)
thank you guys for answering the open questions. What is open still is the question how we can structure the tests, i.e. job group, variables, flavors, etc. See updated description. Maybe others want to comment.
Updated by okurz over 8 years ago
[03/08/2016 13:07:24] okurz: hmm.. jobgroups is a tough one - as far as I see it is 'independent products' and I'd thus see one group per product
[03/08/2016 13:07:35] (argon, krypton, krypton-next)
[03/08/2016 13:08:25] ok, will start with just one group for krypton then and add others later --> https://openqa.opensuse.org/admin/job_templates/23
Updated by okurz over 8 years ago
- Created medium on openqa instance (on lord.arch as well as o3)
sudo -u _openqa-worker /usr/share/openqa/script/client products get distri=opensuse version=* flavor=Krypton-Live arch=x86_64 | grep -q Krypton-Live || sudo -u _openqa-worker /usr/share/openqa/script/client products post distri=opensuse version=* flavor=Krypton-Live arch=x86_64 settings[LIVECD]=1 settings[ISO_MAXSIZE]=999999999
- Created job group "openSUSE Krypton" over web ui
- Created testsuite "krypton-live" with settings as LIVETEST=1
Added medium to job group with testsuite
Configured http://lord.arch:8080/job/krypton-openqa to trigger iso with
/usr/share/openqa/script/client --host https://openqa.opensuse.org \
isos post DISTRI=opensuse VERSION=${version} \
FLAVOR=Krypton-Live BUILD=${build} ARCH=${arch} ISO_URL=${download}${newest_krypton_iso}
using ISO download.
First test on o3 triggered --> t#235811
Updated by okurz over 8 years ago
- Description updated (diff)
current tests triggered (already excluding system update):
isosize tests/installation/isosize.pm
bootloader tests/installation/bootloader.pm
finish_desktop tests/installation/finish_desktop.pm
consoletest_setup tests/console/consoletest_setup.pm
check_console_font tests/console/check_console_font.pm
textinfo tests/console/textinfo.pm
hostname tests/console/hostname.pm
xorg_vt tests/console/xorg_vt.pm
zypper_lr tests/console/zypper_lr.pm
zypper_ref tests/console/zypper_ref.pm
yast2_lan tests/console/yast2_lan.pm
curl_https tests/console/curl_https.pm
glibc_i686 tests/console/glibc_i686.pm
zypper_in tests/console/zypper_in.pm
yast2_i tests/console/yast2_i.pm
vim tests/console/vim.pm
firewall_enabled tests/console/firewall_enabled.pm
sshd tests/console/sshd.pm
mtab tests/console/mtab.pm
consoletest_finish tests/console/consoletest_finish.pm
xterm tests/x11/xterm.pm
kate tests/x11/kate.pm
firefox tests/x11/firefox.pm
firefox_audio tests/x11/firefox_audio.pm
ooffice tests/x11/ooffice.pm
khelpcenter tests/x11/khelpcenter.pm
systemsettings5 tests/x11/systemsettings5.pm
dolphin tests/x11/dolphin.pm
amarok tests/x11/amarok.pm
kontact tests/x11/kontact.pm
reboot_plasma5 tests/x11/reboot_plasma5.pm
desktop_mainmenu tests/x11/desktop_mainmenu.pm
shutdown tests/x11/shutdown.pm
Is there any test that does not make sense to be called in Krypton?
Updated by okurz over 8 years ago
- Copied to action #13246: [opensuse]Add tests for Live-DVD Installer added
Updated by okurz about 8 years ago
OBS projects are in https://build.opensuse.org/project/show/KDE:Medias
Using something like https://github.com/openSUSE/osc-plugin-factory/blob/master/openqa-comments.py or
https://api.opensuse.org/apidocs#89 it should be possible to post comments on the corresponding subproject or packages, e.g.
osc api -X POST "/comments/package/KDE:Medias/openSUSE-Krypton…
the comment content is delivered by the data attribute, e.g. like
osc api --data='foo' -X POST "/comments/package/home:okurz/python-openqa_review"
Updated by okurz about 8 years ago
- Description updated (diff)
- DONE: optional: use for marketing (see #12956#note-1) done with blog article by @luca_b: https://www.dennogumi.org/2016/11/testing-the-untestable/
Updated by okurz about 8 years ago
- Description updated (diff)
- Status changed from In Progress to Resolved
comments are posted on https://build.opensuse.org/package/show/KDE:Medias/openSUSE-Krypton based on the result of the monitor-openqa_job
e.g. by
/opt/openqa-scripts-feature-okurz/monitor-openqa_job || (osc api --data="test failed, see https://openqa.opensuse.org/group_overview/23" -X POST "/comments/package/KDE:Medias/openSUSE-Krypton" ; exit 1)
with which I consider this ticket done. Of course, eventually someone will complain that there are too many comments :-) An approach as done within the python tools of http://github.com/openSUSE/osc-plugin-factory/ should be preferred. AFAICS these tools reuse the existing comment and don't just add a new one but that's something for later.