QA » openQA Project » openQA Tests

Retriggered, should check later: https://openqa.suse.de/tests/957462

As mgriessmeier and me found out the problem is actually the same as described in bsc#1040606 so it is this bug.

Still, the problem is very misleading because not many would expect the ssh daemon in the SUT to be not working when we can not switch to a text terminal within the consoletest_setup test module.

added to current sprint backlog

helpful PRs to get more information about the incompletes were merged meanwhile:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3604
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3600

no obvious incompletes spotted in the last few builds, so lowering priority and setting to feedback

ok so after the latest PR merged we have retriggered a whole medium again for SLE15 tests and couldn't find any related problems but also some of the tests still don't ever progress that far. Also I ran "create_hdd_minimal_base+sdk@zkvm" 20 times without any problems. So I cloned sle-12-SP3-Server-DVD-s390x-Build0473-gnome+proxy_SCC+allmodules@zkvm now to check for further problems if any.

EDIT: Well that experiment failed early with "nothing provides product(SLES) = 12 needed by product:sle-module-adv-systems-management-12-0.s390x" so either all the modules on SCC are now broken or I did something wrong.

ok, when taking a look at the latest SLE15 build we don't find any incompletes. I assume we improved here but can reopen when we see same symptoms.

Seems we are not done here. See https://openqa.suse.de/tests/1269751#step/boot_to_desktop/19

Here we try to select the root-console, i.e. connect to the SUT over SSH, which is reproducibly not working in 2/2 jobs after retriggering in this scenario.

The one working in before is already 14d old so I retriggered an older one to test if that still works or if something in the build caused it:

$ openqa_clone_job_osd --skip-chained-deps 1251541 TEST=okurz_reproduce_reconnect_problem _GROUP=0

https://openqa.suse.de/tests/1269791

died as incomplete, I don't understand the error: https://openqa.suse.de/tests/1269791/file/autoinst-log.txt :(

Seems we have to revisit this beast…

I've debugged this issue further together with nick:

Still sporadic, though really good reproducible on local instance:

1. Successful run:
   $self->run_cmd("! virsh dominfo $vmname | grep -w 'shut off'");
   gets executed multiple times until the return code = 1

2. Incompletes
   $self->run_cmd("! virsh dominfo $vmname | grep -w 'shut off'");
   gets executed once, return code is 0, which is fine, because the machine isn't shut down yet

After this it hangs, see Debug output:

[2018-01-03T15:06:11.0114 CET] [debug] Connection to root@s390p8.suse.de established
[2018-01-03T15:06:11.0248 CET] [debug] Command executed: ! virsh dominfo openQA-SUT-7 | grep -w 'shut off', ret=0
[2018-01-03T15:06:11.0248 CET] [debug] === AFTER READ, rsp: HASH(0x66b2948)
[2018-01-03T15:06:11.0248 CET] [debug] === IS_SHUTDOWN RETURN VALUE: 0
         Stopping Restore /run/initramfs on shutdown...
         Stopping Command Scheduler...
         Stopping SuSEfirewall2 phase 2...
         Stopping User Manager for UID 0...
         Stopping Session 1 of user root.
[  OK  ] Stopped Serial Getty on ttysclp0.
[  OK  ] Stopped Serial Getty on sclp_line0.
[  OK  ] Stopped System Logging Service.
[  OK  ] Stopped OpenSSH Daemon.
[  OK  ] Stopped Command Scheduler.
[  OK  ] Stopped User Manager for UID 0.
[  OK  ] Stopped Restore /run/initramfs on shutdown.
[  OK  ] Stopped Session 1 of user root.
[  OK  ] Removed slice User Slice of root.
         Stopping Login Service...
         Stopping Postfix Mail Transport Agent...
         Stopping Permit User Sessions...
[  OK  ] Stopped /etc/init.d/boot.local Compatibility.
[  OK  ] Removed slice system-serial\x2dgetty.slice.
[  OK  ] Stopped Login Service.
[  OK  ] Stopped Login and scanning of iSCSI devices.
[  OK  ] Stopped target Network is Online.
[  OK  ] Stopped Load kdump kernel and initrd.
[  OK  ] Started Show Plymouth Power Off Screen.
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Stopped target Remote File Systems.
[  OK  ] Stopped target Remote File Systems (Pre).
[  OK  ] Stopped target User and Group Name Lookups.
[  OK  ] Stopped Load kdump kernel early on startup.
[  OK  ] Stopped SuSEfirewall2 phase 2.
[  OK  ] Stopped Postfix Mail Transport Agent.
[  OK  ] Stopped target Network.
         Stopping wicked managed network interfaces...
[  OK  ] Stopped target Host and Network Name Lookups.
         Stopping Name Service Cache Daemon...
[  OK  ] Stopped Name Service Cache Daemon.
[  OK  ] Stopped wicked managed network interfaces.
         Stopping wicked network nanny service...
[  OK  ] Stopped wicked network nanny service.
         Stopping wicked network management service daemon...
[  OK  ] Stopped wicked network management service daemon.
         Stopping wicked DHCPv4 supplicant service...
         Stopping wicked AutoIPv4 supplicant service...
         Stopping wicked DHCPv6 supplicant service...
[  OK  ] Stopped wicked DHCPv6 supplicant service.
[  OK  ] Stopped wicked AutoIPv4 supplicant service.
[  OK  ] Stopped wicked DHCPv4 supplicant service.
[  OK  ] Stopped SuSEfirewall2 phase 1.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Sockets.
[  OK  ] Closed Open-iSCSI iscsid Socket.
[  OK  ] Closed PC/SC Smart Card Daemon Activation Socket.
[  OK  ] Closed Syslog Socket.
[  OK  ] Stopped target Paths.
[  OK  ] Stopped target Slices.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped target System Initialization.
         Stopping Load/Save Random Seed...
[  OK  ] Stopped target Swap.
         Deactivating swap /dev/disk/by-path/ccw-0.0.0000-part2...
[  OK  ] Stopped Update is Completed.
[  OK  ] Stopped Rebuild Journal Catalog.
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped Load Kernel Modules.
         Stopping Update UTMP about System Boot/Shutdown...
[  OK  ] Stopped target Encrypted Volumes.
[  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Stopped Rebuild Hardware Database.
         Stopping LSB: Set Control Program Identification data...
[  OK  ] Deactivated swap /dev/disk/by-uuid/...b3c-d26c-443a-9ba2-964b7360e65d.
[  OK  ] Deactivated swap /dev/vda2.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Deactivated swap /dev/disk/by-path/ccw-0.0.0000-part2.
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped Flush Journal to Persistent Storage.
[  OK  ] Stopped target Local File Systems.
         Unmounting /opt...
         Unmounting /var/tmp...
         Unmounting /var/lib/pgsql...
         Unmounting /var/lib/mariadb...
         Unmounting /var/lib/mailman...
         Unmounting /srv...
         Unmounting /var/cache...
         Unmounting /var/lib/libvirt/images...
         Unmounting /var/crash...
         Unmounting /usr/local...
         Unmounting /var/lib/machines...
         Unmounting /var/lib/mysql...
         Unmounting /run/user/0...
         Unmounting /.snapshots...
         Unmounting /var/spool...
         Unmounting /var/lib/named...
         Unmounting /home...
         Unmounting /tmp...
         Unmounting /boot/zipl...
         Unmounting /var/opt...
         Unmounting /boot/grub2/s390x-emu...
         Unmounting /var/log...
[  OK  ] Unmounted /opt.
[  OK  ] Unmounted /var/tmp.
[  OK  ] Unmounted /var/lib/pgsql.
[  OK  ] Unmounted /var/lib/mariadb.
[  OK  ] Unmounted /var/lib/mailman.
[  OK  ] Unmounted /srv.
[  OK  ] Unmounted /var/cache.
[  OK  ] Unmounted /var/lib/libvirt/images.
[  OK  ] Unmounted /var/crash.
[  OK  ] Unmounted /usr/local.
[  OK  ] Unmounted /var/lib/machines.
[  OK  ] Unmounted /var/lib/mysql.
[  OK  ] Unmounted /run/user/0.
[  OK  ] Unmounted /.snapshots.
[  OK  ] Unmounted /var/spool.
[  OK  ] Unmounted /var/lib/named.
[  OK  ] Unmounted /home.
[  OK  ] Unmounted /tmp.
[  OK  ] Unmounted /var/opt.
[  OK  ] Unmounted /boot/zipl.
[  OK  ] Unmounted /boot/grub2/s390x-emu.
[  OK  ] Unmounted /var/log.
[  OK  ] Stopped File System Check on /dev/disk/by-path/ccw-0.0.0000-part1.
[  OK  ] Removed slice system-systemd\x2dfsck.slice.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped target Local File Systems (Pre).
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped LSB: Set Control Program Identification data.
[  OK  ] Reached target Shutdown.

[2018-01-03T15:06:12.0249 CET] [debug] === ITERATION ASSERT_SHUTDOWN with remaining timeout 59
Use of uninitialized value $args in concatenation (.) or string at /home/os-autoinst/autotest.pm line 250.
[2018-01-03T15:06:12.0249 CET] [debug] === QUERY_ISOTOVIDEO CALLED, CMD: backend_is_shutdown, ARGS: 
[2018-01-03T15:06:12.0249 CET] [debug] === BEFORE SEND, CMD: backend_is_shutdown, JSON: backend_is_shutdown
[2018-01-03T15:06:12.0249 CET] [debug] === BEFORE READ, isotovideo: GLOB(0x1881938)

so, one can see that the issue is not in the testcode, since the while loop in assert_shutdown is iterated again, but then the backend hangs in query_isotovideo('backend_is_shutdown')

18/18 runs successful - no incompletes with PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4107
setting to feedback until PR is merged

https://openqa.suse.de/tests/1371296 successful on o.s.d

I know that this issue was sporadic and one verification run is probably not sufficient, but taking the other ~5 tests i did manually into account, I consider this as resolved
If you see any incompletes on s390-kvm in shutdown again, please reopen