QA » openQA Project » openQA Infrastructure

Needle load time has increased since 09.10.2019 morning, from 3-5 seconds, to 60+ seconds.

This may cause failures in tests with time sensitive screen actions, for example offline upgrades with PATCH=1 which schedule:

migration/reboot_to_upgrade
migration/version_switch_upgrade_target
installation/isosize
installation/bootloader

The needle load time increase makes migration/version_switch_upgrade_target to take over 2 minutes to complete while the VM is rebooting in the background. When the test reaches installation/bootloader, SUT is already in the GRUB menu so it is not able to select upgrade from the installation menu and fails.

The following data is from HA offline migration tests and compare the last failing one vs the last passing one.

offline + SCC migration from 12sp4:¶

[2019-10-09T16:01:52.266 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 14:01:52 (108 s)
vs
[2019-10-09T01:03:01.624 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 23:03:01 (3 s)

offline + SCC migration from 12sp4:¶

[2019-10-09T22:12:39.589 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:12:39 (121 s)
vs
[2019-10-09T00:57:00.555 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 22:57:00 (3 s)

offline from 11sp4:¶

[2019-10-09T18:52:58.316 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 16:52:58 (194 s)
vs
[2019-10-09T01:16:05.424 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 23:16:05 (3 s)

offline from 11sp4:¶

[2019-10-09T18:51:43.165 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 16:51:43 (212 s)
vs
[2019-10-09T01:04:28.904 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 23:04:28 (3 s)

offline from 12sp2-ltss:¶

[2019-10-09T18:44:03.649 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 16:44:03 (190 s)
vs
[2019-10-09T00:55:09.279 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 22:55:09 (3 s)

offline from 12sp2-ltss:¶

[2019-10-09T22:09:06.597 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:09:06 (198 s)
vs
[2019-10-09T00:50:04.276 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 22:50:04 (5 s)

offline from 12sp3-ltss:¶

[2019-10-09T22:13:16.424 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:13:16 (152 s)
vs
[2019-10-09T01:06:02.459 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 23:06:02 (3 s)

offline from 12sp3-ltss:¶

[2019-10-09T22:15:35.895 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:15:35 (121 s)
vs
[2019-10-09T01:07:07.188 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 23:07:07 (3 s)

offline from 12sp4:¶

[2019-10-09T22:19:55.434 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:19:55 (487 s)
vs
[2019-10-09T00:52:54.409 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 22:52:54 (5 s)

offline from 12sp4:¶

[2019-10-09T22:04:26.060 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-09 20:04:26 (97 s)
vs
[2019-10-09T00:57:15.750 CEST] [debug] ||| finished version_switch_upgrade_target migration at 2019-10-08 22:57:15 (3 s)

More logs in: https://openqa.suse.de/tests/overview?distri=sle&version=12-SP5&build=0155%400350&groupid=157