action #10996

[sles][migration] fully patch before upgrading (offline+online)

Added by RBrownSUSE almost 4 years ago. Updated almost 3 years ago.

Status:ResolvedStart date:29/02/2016
Priority:UrgentDue date:
Assignee:mitiao% Done:

100%

Category:Enhancement to existing tests
Target version:openQA Project - Milestone 7
Difficulty:
Duration:

Description

all migrations, including offline and online migrations, should fully patch before upgrading, as mentioned in the documentation

Actually, two different test cases must be covered, from PRD (page 25):

  • All system updates from maintenance channels applied (aka. "fully patched")
  • With minimal necessary patches (only call zypper patch --updatestack-only)

(For online migration see related ticket)

sp2reg.png (18.1 KB) dmaiocchi, 31/03/2016 07:50 am

2132

Checklist

  • sle

Related issues

Related to openQA Tests - action #11000: [Migration] migrations - upgrades with modules Resolved 29/02/2016
Related to openQA Tests - action #15112: [Migration] SLES: minimal necessary patches Resolved 29/11/2016
Blocked by openQA Tests - action #18172: [sle][migration][ha]test fails in zypper_lr consistently Resolved 30/03/2017

History

#1 Updated by RBrownSUSE almost 4 years ago

  • Priority changed from High to Urgent

#2 Updated by RBrownSUSE almost 4 years ago

  • Assignee set to dmaiocchi

#3 Updated by dmaiocchi almost 4 years ago

  • Status changed from New to In Progress

#4 Updated by dmaiocchi almost 4 years ago

after thinking how to do that, i have only one solution, is an sys-admin/QA solution out of the scope of openqa-tests it self.

We should have a mirror/server that make update of the qcow(if manually or automated should be discussed) of the disks(qcow) that need to be updated. (automatically could be packages-conflicts).

With openqa itself, in the test-scenario (migrating before upgrading) technically, imho you have to boot first on the qcow, (no problem) but then you have to reattach the dvd, (this is done with qemu-backend at openqa launch. not during a test)to pass some grub parameter even edit the grub.cfg file to reboot on the disk-images, and i don't know if the disk passed by qemu should be there or not. So this approach sound to me too complicated and even there are too much dependencies that could break the test at beginning.

The first one could be a stable, general solution, but has to have more maintenance, but could have more generality, and scalable, since everyone could access the full-updated images, for running other tests.

I'm open for hints in both direction or new directions.

#5 Updated by dmaiocchi almost 4 years ago

  • Status changed from In Progress to Feedback

#6 Updated by coolo almost 4 years ago

we already have a "Maintenance: Released Updates" job group (actually 2 per SP). You can schedule jobs in there to update the assets - this job group is scheduled daily. This means for migration we always have freshly patched qcows ready

#7 Updated by dmaiocchi almost 4 years ago

checklist{
1. add/integrate updated sles-12-sp1.qcow: WIP
2. add/integrate updated sles-12-sp0.qcow:
3. add/integrate ..

#8 Updated by dmaiocchi almost 4 years ago

if you have a registrated system (sles-12-sp0) and upgrade -> sles-12-sp2. this systems want to contact the registration proxy by default?
in this case, we should add a job on maintenance-group, to create a not registred qcow, otherwise we cannot use on migration scenario for sp2

#9 Updated by dmaiocchi almost 4 years ago

Question: is SP-2 ready for accept scc request from registrated systems?

HOW_TO_REPRODUCE: Take the asset from qa-maintenance job-group, use this in a group, json file :

/usr/share/openqa/script/client --params update-12sp1.json jobs post
{
"ARCH" : "x86_64",
"BETA" : "1",
"BETA_SDK" : "1",
"BETA_WE" : "1",
"BUILD" : "1294",
"BUILD_HA" : "0196",
"BUILD_HA_GEO" : "0055",
"BUILD_SDK" : "0542",
"BUILD_WE" : "0154",
"DESKTOP" : "gnome",
"DISTRI" : "sle",
"DVD" : 1,
"FAKE_SCC_EMAIL" : "openqa@suse.de",
"FAKE_SCC_REGCODE" : "SLE-12-SP2-Server-DVD-x86_64-6gXVQP2tcAcEq3iD",
"FAKE_SCC_REGCODE_HA" : "SLE-12-SP2-Server-DVD-x86_64-6gXVQP2tcAcEq3iD",
"FAKE_SCC_REGCODE_HA_GEO" : "SLE-12-SP2-Server-DVD-x86_64-6gXVQP2tcAcEq3iD",
"FAKE_SCC_REGCODE_WE" : "SLE-12-SP2-Server-DVD-x86_64-6gXVQP2tcAcEq3iD",
"FAKE_SCC_URL" : "http://scc.openqa.suse.de",
"FLAVOR" : "Server-DVD",
"GNOME" : 1,
"HASLICENSE" : 1,
"HDDSIZEGB" : "40",
"NUMDISKS" : 1,
"NOIMAGES" : 1,
"HDD_1" : "SLE-Server-DVD-Updates-12-SP1-x86_64-default_registered_with_updates.qcow2",
"HOST" : "localhost",
"INSTLANG" : "en_US",
"ISO" : "SLE-12-SP2-Server-DVD-x86_64-Build1294-Media1.iso",
"ISO_MAXSIZE" : "4700372992",
"MACHINE" : "64bit",
"NOAUTOLOGIN" : 1,
"PACKAGETOINSTALL" : "x3270",
"QA_HEAD_REPO" : "http://dist.nue.suse.com/ibs/QA:/Head/SLE-12-SP2",
"REPO_0" : "SLE-12-SP2-Server-DVD-x86_64-Build1294-Media1",
"REPO_10" : "SLE-12-SP2-HA-POOL-x86_64-Build0196-Media1",
"REPO_11" : "SLE-12-SP2-HA-POOL-x86_64-Build0196-Media1.license",
"REPO_12" : "SLE-12-SP2-HA-GEO-POOL-s390x-x86_64-Build0055-Media1",
"REPO_13" : "SLE-12-SP2-HA-GEO-POOL-s390x-x86_64-Build0055-Media1.license",
"REPO_14" : "SLE-12-SP2-WE-POOL-x86_64-Build0154-Media1",
"REPO_15" : "SLE-12-SP2-WE-POOL-x86_64-Build0154-Media1.license",
"REPO_7" : "SLE-12-SP2-Server-POOL-x86_64-Build1294-Media1",
"REPO_8" : "SLE-12-SP2-SDK-POOL-x86_64-Build0542-Media1",
"REPO_9" : "SLE-12-SP2-SDK-POOL-x86_64-Build0542-Media1.license",
"SCC_CERT" : null,
"SCC_EMAIL" : "openqa@suse.de",
"SCC_REGCODE" : "SLE-12-SP2-Server-DVD-x86_64-6gXVQP2tcAcEq3iD",
"SCC_URL" : "http://scc.openqa.suse.de",
"SHUTDOWN_NEEDS_AUTH" : "1",
"SLENKINS_TESTSUITES_REPO" : "http://download.suse.de/ibs/Devel:/SLEnkins:/testsuites/SLE_12_SP2/",
"TEST" : "migration_offline_sle12sp1-update",
"UPGRADE" : "1",
"VERSION" : "12-SP2",
"VNC" : "91"
}

-> the test fail, when contacting the scc server, even in the skip registration.

Solution/Workaround:

1) Add manually the repos from job-maintance test. (new-test for that). this isn't a registration but the repos can be fetched for update.
-> Not nice solution, because will expose the sles-repos on github, and is an hardcoded one.

2) .. ?

#10 Updated by dmaiocchi almost 4 years ago

Conclusion:

after discussion with dgutu, and tests , an activated/registrated qcow cannot migrate to sp2 at moment. We get the error attached.

#11 Updated by dgutu almost 4 years ago

  • Related to action #11000: [Migration] migrations - upgrades with modules added

#12 Updated by dmaiocchi over 3 years ago

  • Assignee deleted (dmaiocchi)

#13 Updated by okurz over 3 years ago

  • Description updated (diff)
  • Status changed from Feedback to New

#14 Updated by okurz over 3 years ago

  • Assignee set to dgutu

As discussed with marita:
dgutu, there are many migration scenarios already present in openqa.suse.de done by mitiao, visible in http://openqa.suse.de/group_overview/29
Please crosscheck with mitiao on the current status, what is missing, what is left to do and update both this issue and the related one #11000
I assume that "only patch update stack" is not covered yet but should be trivial to add to the test code. Please also triage the current status of the migration jobs and propose which are stable enough to go into acceptance group already and which not. Also consider the ones that already catch product issues and label accordingly.

#15 Updated by okurz over 3 years ago

@dgutu, mitiao: Please update this ticket to the current status. If I am not mistaken SLE11SP4 migration is still missing except for offline migration in the acceptance test group

#16 Updated by mitiao over 3 years ago

online migration tests already covered fully patch and minimal necessary patches before migration.
'zypper patch --updatestack-only' would be executed by yast2/zypper migration if the system was not fully patched.

fully patch before migration
https://openqa.suse.de/tests/498586#step/zypper_patch/1

minimal patch before migration
zypper: https://openqa.suse.de/tests/498083#step/zypper_migration/1
yast : https://openqa.suse.de/tests/497749#step/yast2_migration/12

#17 Updated by dgutu over 3 years ago

Thank you @mitiao for you comment, that explains most of the requirements for online migration.
I will focus on /offline/ one and what I've found until now is that:
1. On SLES-12-SP0 zypper should be upgraded to be able to use "zypper patch --updatestack-only"
if I add DVD1 with SP2 then I get packages conflict. I can use workaround magic.

#18 Updated by okurz over 3 years ago

dgutu wrote:

if I add DVD1 with SP2 then I get packages conflict. I can use workaround magic.

why should you add this DVD? I guess the idea is to boot into the system, call the update command and then restart into the offline migration which will do the upgrade to SP2.

I need to patch before upgrade, that's the requirements in this task. I can't use zypper patch with option "--updatestack-only" in SLES12SP0
and I consider the source of offline-migration DVD with SLES12SP2. I'm sleepy?

#19 Updated by dgutu over 3 years ago

okurz wrote:

dgutu wrote:

if I add DVD1 with SP2 then I get packages conflict. I can use workaround magic.


why should you add this DVD? I guess the idea is to boot into the system, call the update command and then restart into the offline migration which will do the upgrade to SP2.


I need to patch before upgrade, that's the requirements in this task. I can't use zypper patch with option "--updatestack-only" in SLES12SP0

and I consider the source of offline-migration DVD with SLES12SP2. I'm sleepy?

Or let me explain that I'm not reinventing the tests :), I'm re-using the existing offline migration test which works in Acceptance Group.

#20 Updated by okurz over 3 years ago

I guess it would be easier to boot into the existing test, update and reboot into offline migration test, also, reusing existing tests and scenarios, just with an additional step for this additional scenario. We want to preserve the existing scenarios anyway, not overwrite them.

#21 Updated by dgutu over 3 years ago

okurz wrote:

I guess it would be easier to boot into the existing test, update and reboot into offline migration test, also, reusing existing tests and scenarios, just with an additional step for this additional scenario. We want to preserve the existing scenarios anyway, not overwrite them.

So you mean the order could be this:
1. Register system during installation (SLES-12-SP0 and SLES-12-SP1)
2. Install all updates/patches
3. Save/publish HDD
4. Start offline upgrade from

Am I right?

#22 Updated by RBrownSUSE over 3 years ago

dgutu wrote:

So you mean the order could be this:

1. Register system during installation (SLES-12-SP0 and SLES-12-SP1)

2. Install all updates/patches

3. Save/publish HDD

4. Start offline upgrade from


Am I right?

No, because a registration includes a database entry inside SCC for the 12-SP0 or 12-SP1 machine

As soon as that ONE installation is upgraded, it becomes an SP2 database entry and cannot be re-used as an 12-SP0 or 12-SP1 system

That would make the reuse of the image impossible

Therefore the only option is to

  1. Install without registration 12-SP0 or 12-SP1
  2. Reuse the image in lots of different scenarios, boot the reused image
  3. Register the 12-SP0 or 12-SP1 machine
  4. Patch it
  5. Reboot into 12-SP2 media and start offline/online migration

#23 Updated by dgutu over 3 years ago

RBrownSUSE wrote:

dgutu wrote:

So you mean the order could be this:

1. Register system during installation (SLES-12-SP0 and SLES-12-SP1)

2. Install all updates/patches

3. Save/publish HDD

4. Start offline upgrade from


Am I right?


No, because a registration includes a database entry inside SCC for the 12-SP0 or 12-SP1 machine


As soon as that ONE installation is upgraded, it becomes an SP2 database entry and cannot be re-used as an 12-SP0 or 12-SP1 system


That would make the reuse of the image impossible


Therefore the only option is to


  1. Install without registration 12-SP0 or 12-SP1
  2. Reuse the image in lots of different scenarios, boot the reused image
  3. Register the 12-SP0 or 12-SP1 machine
  4. Patch it
  5. Reboot into 12-SP2 media and start offline/online migration

I have reused and modified some tests which are used in migration online.
So steps are the follows:
1. Boot clear/unregistered HDD image with sles12sp0 & sles12sp1
2. Register them against Proxy-SCC
3. Get patches
4. De-register
5. Generate HDD-patched
6. Use them for offline migrations.
Some examples:

http://crocodile.qa.suse.cz/tests/3990
http://crocodile.qa.suse.cz/tests/3989

Of course this will like one function :) but as a draft I think it looks good.
What about SLE11-SP4 ? The same logic?

#24 Updated by RBrownSUSE over 3 years ago

SLE11-SP4, same logic should work for offline migrations, yes

will not work for online migrations as 'step 7' of the online migration is "boot and register" before starting the upgrade, and it takes 48 hours+ after registering a SLE 11 machine before it appears in SCC, therefore any SCC-bound tests, such as online migration will fail.

But the online migrators already know this - just stating it here for completeness, and it helps explain why testing this properly in offline migration is so important :)

#25 Updated by dgutu over 3 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

#26 Updated by maritawerner over 3 years ago

What is the current status here?

#27 Updated by dgutu over 3 years ago

maritawerner wrote:

What is the current status here?

The status is the following:
1. On the server ain.arch.suse.de I have created the test cases for all migration scenarios
that we currently have on main openqa
2. I need at least a couple of days to run the tests against SP3

Some examples to check, sles11-sp4 migration (offline) to sles12-sp3
http://ain.arch.suse.de/tests/1055 - patching
http://ain.arch.suse.de/tests/1055 - migration

#28 Updated by maritawerner over 3 years ago

  • Target version changed from Milestone 2 to Milestone 4

#29 Updated by maritawerner about 3 years ago

  • Related to action #15112: [Migration] SLES: minimal necessary patches added

#30 Updated by maritawerner about 3 years ago

  • Description updated (diff)

#31 Updated by dgutu about 3 years ago

  • % Done changed from 30 to 80

#32 Updated by maritawerner about 3 years ago

  • Subject changed from migrations - fully patch before upgrading to [Migration] migrations - fully patch before upgrading

#33 Updated by dgutu about 3 years ago

  • Target version changed from Milestone 4 to Milestone 5

You may take a look at results like here - http://corr.arch.suse.de/tests/overview?build=0209&distri=sle&version=12-SP3&groupid=2
I will add migration with addons and once its passes the task will be closed as resolved.

#34 Updated by dgutu about 3 years ago

I still have 2 issue to resolve before creating a pull-request, this are:
1. The zypper_lr gets confused, i declare that I need to register but in fact I just need to use the update channels
last run on autoupgrade_sles11sp4 - http://corr.arch.suse.de/tests/659
2. The testcase migration_zdup_offline_sle12sp2 fails to mount the installation media
last run - http://corr.arch.suse.de/tests/660#step/zdup/14
This happens only in 'my' environment so o.s.d is just on this step.

#35 Updated by dgutu about 3 years ago

dgutu wrote:

I still have 2 issue to resolve before creating a pull-request, this are:

1. The zypper_lr gets confused, i declare that I need to register but in fact I just need to use the update channels

last run on autoupgrade_sles11sp4 - http://corr.arch.suse.de/tests/659

  • This one was resolved by unset the SCC_REGISTRATION variable after patching
  1. The testcase migration_zdup_offline_sle12sp2 fails to mount the installation media
    last run - http://corr.arch.suse.de/tests/660#step/zdup/14
    This happens only in 'my' environment so o.s.d is just on this step.
  • Today I've tried newer qemu bios, o.s.d uses much newer bios version then on mine so I've tried also newest one but without luck. Giving a try with manual testing.

#36 Updated by dgutu about 3 years ago

dgutu wrote:

  1. The testcase migration_zdup_offline_sle12sp2 fails to mount the installation media
    last run - http://corr.arch.suse.de/tests/660#step/zdup/14
    This happens only in 'my' environment so o.s.d is just on this step.

  • Today I've tried newer qemu bios, o.s.d uses much newer bios version then on mine so I've tried also newest one but without luck.
    Giving a try with manual testing.
  • No such issue on manual testing, I'm creating the PR.

#37 Updated by okurz about 3 years ago

please reference PR and update ticket at least once a week. That should not be to hard for a ticket that is considered "urgent".

#39 Updated by dgutu about 3 years ago

  1. I've faced a situation when one test in migration needs to registration against SCC so right now I'm adapting the patching test to not set registration variable if the test mentioned is not in place.
  2. Also because of registration variable zypper_lr test fails, so I'm working on this part also.

#40 Updated by RBrownSUSE about 3 years ago

How are you going to patch WITHOUT registration to SCC?

I think all of the scenarios probably need registration, because all the supported scenarios need patching, and patching needs registration to provide the repositories

or am I misunderstanding that dependency chain?

#41 Updated by dgutu about 3 years ago

RBrownSUSE wrote:

How are you going to patch WITHOUT registration to SCC?


I think all of the scenarios probably need registration, because all the supported scenarios need patching, and patching needs registration to provide the repositories


or am I misunderstanding that dependency chain?

Yes, a bit of misunderstanding :) Could be my fault.
I do use the scc registration while patching the hdd and then I need to unset the registration variable, otherwise it will
try to register the system during upgrade. There is a testcase migration_offline_sle12sp1_smt which need the registration even on the step
of upgrading. Hope that explains.

#42 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 5 to Milestone 6

#43 Updated by okurz almost 3 years ago

  • Subject changed from [Migration] migrations - fully patch before upgrading to [Migration] fully patch before upgrading (offline+online)

#44 Updated by mitiao almost 3 years ago

failed at https://openqa.suse.de/tests/846489#step/patch_before_migration/45
looks like it is a issue about registration with addons in patch_before_migration.pm

#45 Updated by mitiao almost 3 years ago

created a ticket to solve it:
https://progress.opensuse.org/issues/18224

#46 Updated by dgutu almost 3 years ago

  • Priority changed from Urgent to High

The patching is working fine for x86_64(https://openqa.suse.de/tests/871908) and s390x (https://openqa.suse.de/tests/872179)
More work is need for ppc64le arch, related ticket is https://progress.opensuse.org/issues/18172

#47 Updated by RBrownSUSE almost 3 years ago

  • Priority changed from High to Urgent

I still believe this to be urgent - one architecture being blocked is makes it just as important as previously. We need to test migrations compliant to the product requirements documentation.

#48 Updated by RBrownSUSE almost 3 years ago

  • Blocked by action #18172: [sle][migration][ha]test fails in zypper_lr consistently added

#49 Updated by okurz@suse.de almost 3 years ago

I agree. Keep in mind that dgutu is in holiday for the next days so someone
else might want to take over

#50 Updated by maritawerner almost 3 years ago

  • Subject changed from [Migration] fully patch before upgrading (offline+online) to [sles][migration] fully patch before upgrading (offline+online)
  • Target version changed from Milestone 6 to Milestone 7

#51 Updated by dgutu almost 3 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from dgutu to mitiao

As discussed on the call, Wei Jang please close this one if you consider we covered everything.

#52 Updated by mitiao almost 3 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 80 to 100

I confirmed that we have done the fully patch and even minimal patch before migration both for offline and online.
This feature is working for online/offline migration tests on different archs including x86_64, ppc64le and s390x.
And it would work for aarch64 as well if migration on aarch64 complete.
Thanks.

Also available in: Atom PDF