action #108692
closedcoordination #108797: [qe-core] Leap setup 15.4 openqa for updates
[qe-core] Leap offline migration - do not skip releases (15.0 -> 15.1 -> 15.2 -> 15.3)
Added by lkocman over 2 years ago. Updated over 2 years ago.
0%
Description
Hello team,
this is a conclusion that we came up with Fabian and Santi in https://bugzilla.opensuse.org/show_bug.cgi?id=1196549
We as a community seem to see a value in keeping even unsupported migration scenarios. Migration over several releases seem to be highly appreciated by community and helps to advertise that we're a rock solid distro that doesn't go through rapid changes.
https://en.opensuse.org/SDB:System_upgrade recommends that user of 15.2 should always migrate to 15.3 prior he migrates to 15.4. However that's not what our migration suite does. We're migrating from let's say 15.0 directly to 15.4 and this has stopped working with 15.3 as we did implement a gpg key loopkup in /repodata which was released as maintenance update to I believe SP2/SP3 but not to older releases as they were EOL.
This request is to ensure that we do migrations without skipping the releases, or migrate at least to 15.2 + full updates first (at that point in time systems were able to migrate to 15.3 which is where the issue has started to appear)
I have to admit that it's quite expensive effort as we keep testing migration scenarios from 42.3 to the recent 15.4.
But it has to be done otherwise we can't get over boo#1196549 and there would be no longer a reason to keep broken test migration test suites.
Thank you
Acceptance Criteria¶
- Migration tests are created for the different combinations, ending in an upgrade to 15.4
Remarks¶
gnome, kde, cryptlvm need those 4 upgrade tests
15.0 -> 15.1 -> 15.2 -> 15.3 -> 15.4
15.1 -> 15.2 -> 15.3 -> 15.4
15.2 -> 15.3 -> 15.4
15.3 -> 15.4
Meaning effectively 5 upgrade jobs where all can be chained one to another
Files
| screenshot-2022-05-16-11_16_59.jpg (55.3 KB) screenshot-2022-05-16-11_16_59.jpg | chained migration from 15.1 to 15.4 | mloviska, 2022-05-16 09:20 |
Updated by szarate over 2 years ago
- Subject changed from Leap migration - do not skip releases (15.0 -> 15.1 -> 15.2 -> 15.3) to [qe-core] Leap migration - do not skip releases (15.0 -> 15.1 -> 15.2 -> 15.3)
Updated by szarate over 2 years ago
- Related to action #107620: [qe-core] Please add upgrade test from Leap 15.3 to Leap 15.4 added
Updated by fcrozat over 2 years ago
I would only test version => version+2 at maximum, to reduce test matrix. Fedora is doing the same
Updated by lkocman over 2 years ago
An idea from Dominique. Utilize net-install image which has already the latest zypper to the date. The problem with our migration is that tools that do migration do not have required fixes in. I think he mentioned that TW uses it in this way for migration from old releases.
Updated by dimstar over 2 years ago
lkocman wrote:
An idea from Dominique. Utilize net-install image which has already the latest zypper to the date. The problem with our migration is that tools that do migration do not have required fixes in. I think he mentioned that TW uses it in this way for migration from old releases.
Right, on TW, from 'old' releases we only do 'offline upgrades' (i.e. not from a running, old system) using net-installer and dvd-installer, but not zdup.
This allows us to test extra old to current tumbleweed (we go back to 42.1 at the moment) (we tested zdup as long as we could, but TW changed compression payload of the RPM packages and those old distros could not unpack the new RPM format)
of course for the supported scenarios thee zdup cases should stay valid and functioning; the offline method as only method should only be considered on old platforms (and only if zdup cannot be used reasonably easy)
Updated by szarate over 2 years ago
- Tags changed from to-be-refined, qe-core-april-sprint to qe-core-april-sprint
- Description updated (diff)
Updated by szarate over 2 years ago
- Category changed from Refactor/Code Improvements to Bugs in existing tests
Updated by szarate over 2 years ago
- Category changed from Bugs in existing tests to Enhancement to existing tests
Updated by szarate over 2 years ago
- Description updated (diff)
- Start date deleted (
2022-03-21)
Updated by szarate over 2 years ago
- Sprint set to QE-Core: April Sprint (Apr 13 - May 11)
Updated by openqa_review over 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: zdup-Leap-15.0-gnome
https://openqa.opensuse.org/tests/2316424#step/prepare_system_for_update_tests/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by szarate over 2 years ago
- Sprint changed from QE-Core: April Sprint (Apr 13 - May 11) to QE-Core: May Sprint (May 11 - Jun 08)
Updated by szarate over 2 years ago
Figure out the availability of the repositories for eol'd distros
Updated by mloviska over 2 years ago
szarate wrote:
Figure out the availability of the repositories for eol'd distros
Yup, they are present in http://download.opensuse.org/update/leap/, starting with 42.3
Updated by mloviska over 2 years ago
- Status changed from Workable to In Progress
- Assignee set to mloviska
Updated by szarate over 2 years ago
- Related to coordination #108797: [qe-core] Leap setup 15.4 openqa for updates added
Updated by mloviska over 2 years ago
As of now, I have managed to successfully test migration from leap15.1 ⇾ 15.2 ⇾ 15.3 ⇾ 15.4. For some reason, the console switch time-outs in 15.1 job in setup_zdup after migration from 15.0 in my openqa local instance. Manually, it works fine.
You if @lkocman and @szarate agree, I propose to reduce test case schedule for the chained runs, due to time constrains. Otherwise, the whole test would take too long, and I guess we do not need to run package test modules in the intermediate jobs. It should be enough to run full package testing on the last job, e.g. 15.4 or 15.3 (in 15.3 testing).
Thus, the schedule should look like for 15.{1,2,3} as below. For the 15.4 the schedule stays as it was before.
schedule:
- installation/setup_zdup
- installation/install_service
- installation/zdup
- installation/post_zdup
- boot/boot_to_desktop
- installation/opensuse_welcome
- shutdown/shutdown
I think specifically from older releases, it's perfectly fine to reduce scope. So go ahead Martin
Full test suite shall really be just on the currently supported one (e.g. 15.3 -> 15.4 now). As previously mentioned shortcut to simply directly migrate to 15.2, install all updates (specifically latest zypper) and then migrated to the currently developed release (15.4) is also perfectly fine.
Updated by lkocman over 2 years ago
If we reduce scope we can also close this particular issue https://bugzilla.opensuse.org/show_bug.cgi?id=1197359
Updated by mloviska over 2 years ago
leap15.3 to leap15.4¶
- opensuse-15.4-NET-x86_64-Build222.1-zdup-Leap-kde-15.3to15.4
- opensuse-15.4-NET-x86_64-Build222.1-zdup-Leap-kde-15.3to15.4
This works without any issues, amarok and firefox_audio might not be directly related failures of upgrade procedure.
leap15.2 to leap15.4¶
As per https://bugzilla.opensuse.org/show_bug.cgi?id=1196549#c6 installing updates before running upgrade ( I have updated the image before this run ) works fine
chained migration from leap15.1 -> leap15.2 -> leap15.4¶
- opensuse-15.4-NET-x86_64-Build222.1-zdup-Leap-gnome-15.2to15.4_chained_migration_15.1@64bit
- opensuse-15.4-NET-x86_64-Build222.1-zdup-Leap-kde-15.2to15.4_chained_migration_15.1@64bit
Works fine for both kde and gnome flavors
chained migration from leap15.0 -> leap15.2 -> leap15.4¶
Revealed two problems, the gnome migration chained job failed in my openQA instance, seems like a performance issue and maybe related to below messages from PackageKit (not sure). Manual test went fine though.
susetest login: [ 36.064145] PackageKit[2294]: search-file transaction /11_eecdeacb from uid 1000 finished with success after 1305ms
[ 37.484989] PackageKit[2294]: search-file transaction /12_babeeead from uid 1000 finished with success after 1415ms
[ 38.764573] PackageKit[2294]: search-file transaction /13_cbdadcec from uid 1000 finished with success after 1274ms
[ 39.275347] dbus-daemon[823]: [system] Failed to activate service 'org.bluez': timed out (service_start_timeout=25000ms)
[ 39.276338] pulseaudio[2116]: E: [pulseaudio] bluez5-util.c: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[ 39.988575] PackageKit[2294]: search-file transaction /14_cbadbaab from uid 1000 finished with success after 1219ms
[ 41.203013] PackageKit[2294]: search-file transaction /15_eabacbdb from uid 1000 finished with success after 1209ms
[ 42.397529] PackageKit[2294]: search-file transaction /16_deebaade from uid 1000 finished with success after 1189ms
[ 43.574572] PackageKit[2294]: search-file transaction /17_cbaadcdd from uid 1000 finished with success after 1172ms
In case of kde, there is a problem when updating to leap15.2 from leap15.0 ZYPPER_EXIT_INF_RPM_SCRIPT_FAILED.
[ 989.530582] [RPM][29992]: Transaction ID 62840732 started
[ 989.599247] [RPM][29992]: erase sddm-0.17.0-lp150.8.1.x86_64: success
[ 989.613054] useradd[29995]: failed adding user 'sddm', exit code: 9
[ 989.763129] [RPM][29992]: install sddm-0.17.0-lp150.9.6.1.x86_64: success
[ 989.769151] [RPM][29992]: erase sddm-0.17.0-lp150.8.1.x86_64: success
[ 989.770640] [RPM][29992]: install sddm-0.17.0-lp150.9.6.1.x86_64: success
[ 989.771318] [RPM][29992]: Transaction ID 62840732 finished: 0
.
.
.
[ 1131.470476] [RPM][878]: install libqt5-qtwebengine-5.10.1-lp150.3.4.2.x86_64: success
[ 1131.473261] [RPM][878]: scriptlet %postun(libqt5-qtwebengine-5.10.1-lp150.2.1.x86_64) failure: 2
[ 1131.487814] [RPM][878]: erase libqt5-qtwebengine-5.10.1-lp150.2.1.x86_64: success
[ 1131.491270] [RPM][878]: install libqt5-qtwebengine-5.10.1-lp150.3.4.2.x86_64: success
[ 1131.493408] [RPM][878]: 0 elements failed, 1 scripts failed
[ 1131.494968] [RPM][878]: Transaction ID 628407bc finished: 0
Running full chain migration scenario has another downside and that it takes a lot of time to execute even though I have stripped the test modules for bare minimum in the intermediate jobs and run the full scope test suite only at the last job aka leap15.4.
With respect to above, I would vote for Frederic's suggestion, maybe additionally have full chain test for minimal (desktop?) installations.
Updated by mloviska over 2 years ago
I have updated the Leap15.2 assets and the tests are passing the previous repo failure, however KDE fails with authentication problem, any ideas ?
- opensuse-15.4-NET-x86_64-Build236.1-zdup-Leap-15.2-gnome@64bit
- opensuse-15.4-NET-x86_64-Build236.1-zdup-Leap-15.2-kde@64bit
I would also like to know what to do with then older zdup test suites than 15.2 as we cannot fix them. Would you like to chain? If so should we keep gnome|kde or use minimal desktop story ?
Updated by mloviska over 2 years ago
mloviska wrote:
I have updated the Leap15.2 assets and the tests are passing the previous repo failure, however
KDEfails with authentication problem, any ideas ?
May 23 08:58:41.443455 susetest dbus-daemon[613]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.56' (uid=0 pid=1738 comm="pkcon refresh ")
May 23 08:58:41.452548 susetest systemd[1]: Starting PackageKit Daemon...
May 23 08:58:41.475770 susetest PackageKit[1743]: daemon start
May 23 08:58:41.699846 susetest dbus-daemon[613]: [system] Successfully activated service 'org.freedesktop.PackageKit'
May 23 08:58:41.701853 susetest systemd[1]: Started PackageKit Daemon.
May 23 08:58:41.710746 susetest plasmashell[1302]: plasma-pk-updates: Is net mobile: false
May 23 08:58:41.710780 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.711036 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.714625 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.715290 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.715345 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.715544 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.715661 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.745329 susetest PackageKit[1743]: uid 1000 is trying to obtain org.freedesktop.packagekit.system-sources-refresh auth (only_trusted:0)
May 23 08:58:41.715681 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.764936 susetest PackageKit[1743]: uid 1000 failed to obtain auth
May 23 08:58:41.715721 susetest plasmashell[1302]: plasma-pk-updates: Daemon changed
May 23 08:58:41.775703 susetest polkitd[622]: Registered Authentication Agent for unix-process:1738:14960 (system bus name :1.58 [/usr/bin/pkttyagent --notify-fd 10 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale POSIX)
May 23 08:58:41.720255 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.813052 susetest PackageKit[1743]: uid 0 is trying to obtain org.freedesktop.packagekit.system-sources-refresh auth (only_trusted:0)
May 23 08:58:41.720274 susetest plasmashell[1302]: plasma-pk-updates: Checking for updates
May 23 08:58:41.825327 susetest PackageKit[1743]: uid 0 obtained auth for org.freedesktop.packagekit.system-sources-refresh
May 23 08:58:41.731409 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.731677 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.871268 susetest plasmashell[1302]: plasma-pk-updates: Transaction status changed: "waiting-for-auth" "(101%)"
May 23 08:58:41.874194 susetest plasmashell[1302]: plasma-pk-updates: Transaction status changed: "waiting-for-auth" "(101%)"
May 23 08:58:41.884515 susetest plasmashell[1302]: PK error: "Failed to obtain authentication." type: "not-authorized"
May 23 08:58:41.884768 susetest plasmashell[1302]: plasma-pk-updates: Transaction "/1_aeecedcc" finished with status "failed" in 0 seconds
May 23 08:58:41.884810 susetest plasmashell[1302]: plasma-pk-updates: Cache transaction didn't finish successfully
May 23 08:58:41.884853 susetest plasmashell[1302]: plasma-pk-updates: Last check successful: false
May 23 08:58:41.885475 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.885825 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.886242 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.886300 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.886314 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 08:58:41.886434 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
.
.
.
May 23 09:00:29.906617 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:29.913957 susetest systemd[1]: Starting PackageKit Daemon...
May 23 09:00:30.067042 susetest systemd[1]: Started PackageKit Daemon.
May 23 09:00:30.089482 susetest PackageKit[1978]: new update-packages transaction /1_cebdccce scheduled from uid 1000
May 23 09:00:30.082987 susetest plasmashell[1302]: plasma-pk-updates: Is net mobile: false
May 23 09:00:30.083008 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083088 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083125 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083146 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083164 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083190 susetest plasmashell[1302]: plasma-pk-updates: Is net online: true
May 23 09:00:30.083212 susetest plasmashell[1302]: plasma-pk-updates: Daemon changed
May 23 09:00:30.086917 susetest plasmashell[1302]: plasma-pk-updates: Transaction status changed: "wait" "(101%)"
May 23 09:00:30.115781 susetest plasmashell[1302]: plasma-pk-updates: Transaction status changed: "setup" "(101%)"
May 23 09:00:34.256060 susetest plasmashell[1302]: plasma-pk-updates: Transaction status changed: "dep-resolve" "(101%)"
May 23 09:00:34.595394 susetest plasmashell[1302]: plasma-pk-updates: Package updating: "MozillaFirefox;91.9.0-150200.152.37.3;x86_64;repo-sle-update" , info: "installing"
May 23 09:00:34.598511 susetest plasmashell[1302]: plasma-pk-updates: Package updating: "MozillaFirefox-translations-common;91.9.0-150200.152.37.3;x86_64;repo-sle-update" , info: "installing"
May 23 09:00:34.601482 susetest plasmashell[1302]: plasma-pk-updates: Package updating: "cups;2.2.7-150000.3.29.1;x86_64;repo-sle-update" , info: "installing"
Seems like after reload it started to work...
I would also like to know what to do with then older zdup test suites than 15.2 as we cannot fix them. Would you like to chain? If so should we keep
gnome|kdeor use minimal desktop story ?
Updated by mloviska over 2 years ago
- Related to action #111467: [qe-core] opensuse-15.2-aarch64-GM-Updated-gnome@aarch64.qcow2 gone? added
Updated by mloviska over 2 years ago
Updated by mloviska over 2 years ago
- Related to action #91554: [qem][leap][opensuse][qe-core] Update GM images before running upgrade tests added
Updated by mloviska over 2 years ago
- Status changed from In Progress to Feedback
Add leap15.3 upgrade and distri upgrade scenarios.
https://github.com/os-autoinst/opensuse-jobgroups/pull/141
As of now, I am setting the ticket to feedback in order to get more information how to proceed.
Updated by lkocman over 2 years ago
The agreement was to drop speficically zdup migration scenarios for unsupported releases.
We would keep it only 15.2+
Offline media migration and others would still be present as these do not suffer from missing zypper gpg key import feature.
Updated by mloviska over 2 years ago
Updated by mloviska over 2 years ago
Workwise finished, unfortunately, the last PR got merged day after last build of Leap 15.4. In order to make o3 VRs, I can add zdup jobs to Updates test group.
Updated by szarate over 2 years ago
- Subject changed from [qe-core] Leap migration - do not skip releases (15.0 -> 15.1 -> 15.2 -> 15.3) to [qe-core] Leap offline migration - do not skip releases (15.0 -> 15.1 -> 15.2 -> 15.3)
zdup is done, rest of the tests are missing.
Martin to add the zdup tests to the update jobgroup
Updated by szarate over 2 years ago
- Related to action #112169: [qe-core] test fails in zdup - ls: cannot access /dev/disk/by-label added
Updated by szarate over 2 years ago
- Sprint changed from QE-Core: May Sprint (May 11 - Jun 08) to QE-Core: June Sprint (Jun 08 - Jul 06)
Updated by mloviska over 2 years ago
Updated by mloviska over 2 years ago
only GNOME scenarios: https://bugzilla.opensuse.org/show_bug.cgi?id=1200526
Updated by mloviska over 2 years ago
- Status changed from Feedback to Resolved