action #25638: [sles][functional][s390x] test fails in shutdown: VNC stall detected, needs to be investigated - openQA Tests - openSUSE Project Management Tool

Actions

Copy link

action #25638

closed

[sles][functional][s390x] test fails in shutdown: VNC stall detected, needs to be investigated

Added by mgriessmeier over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

mgriessmeier

Category:

Bugs in existing tests

Target version:

openQA Project - Milestone 11

Start date:

2017-09-28

Due date:

2017-10-25

% Done:

Estimated time:

Difficulty:

Description

Observation¶

openQA test in scenario sle-15-Leanos-DVD-s390x-create_hdd_gnome_s390x@s390x-kvm fails in
shutdown (originally: shutdown )

shows similar symptoms as #20022 at least in regards of VNC stalls

Reproducible¶

Fails since (at least) Build 278.1

Steps to reproduce¶

clone test "create_hdd_textmode@s390x-kvm"

Expected result¶

We never executed shutdown successfully in any s390x tests but we try it now.

Further details¶

Always latest result in this scenario: latest (old), now latest

Related issues 6 (0 open — 6 closed)

Related to openQA Tests - action #20022: [sle][functional][zkvm][s390] incomplete test due to socket does not exist. Probably your backend instance could not start or died

Resolved

mgriessmeier

2017-06-23

2017-10-25

Actions

Related to openQA Tests - action #18936: [tools][sles][functional] Enable 3 stress acceptance on s390x

Resolved

riafarov

2017-05-04

2017-11-08

Actions

Related to openQA Tests - action #13216: [sles][functional][s390x] Run extratest on s390x

Resolved

riafarov

2017-03-02

Actions

Related to openQA Project - action #26886: [tools][s390x-kvm] investigate and improve 'assert_shutdown' function in testapi

Resolved

okurz

2017-10-19

Actions

Blocks openQA Tests - action #23406: [sle][functional]Use single test suite for create_hdd_gnome on all architectures (and downstream jobs)

Resolved

riafarov

2017-08-16

2017-10-25

Actions

Copied to openQA Tests - action #26914: [sle][functional][s390x][s390x-kvm] s390x-kvm does never exit from "assert_shutdown" but zkvm works -> investigate how the machines differ, maybe problem on s390p8?

Resolved

okurz

2017-09-28

Actions

Copy link

Updated by mgriessmeier over 6 years ago

Related to action #20022: [sle][functional][zkvm][s390] incomplete test due to socket does not exist. Probably your backend instance could not start or died added

Actions

Copy link

Updated by mgriessmeier over 6 years ago

Blocks action #23406: [sle][functional]Use single test suite for create_hdd_gnome on all architectures (and downstream jobs) added

Actions

Copy link

Updated by mgriessmeier over 6 years ago

Due date set to 2017-10-11

Actions

Copy link

Updated by riafarov over 6 years ago

VNC stalls in other test suites too:
https://openqa.suse.de/tests/1193329
https://openqa.suse.de/tests/1193329

EDIT (okurz): These two should be ignored for the case of this ticket, they do not fail in shutdown

Actions

Copy link

Updated by mgriessmeier over 6 years ago

not worked on that one in particular but https://github.com/os-autoinst/os-autoinst/pull/862 might also help here if anyone wants to investigate further

Actions

Copy link

Updated by riafarov over 6 years ago

PR with fix of review comment to be able to merge it: https://github.com/os-autoinst/os-autoinst/pull/864

Actions

Copy link

Updated by okurz over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by okurz over 6 years ago

did not complete in sprint 1. main reason: spontaneous packaging training which we were not aware of in before. we have the PR which should improve user feedback a lot and this is definitely possible in the next sprint 2.

Actions

Copy link

Updated by okurz over 6 years ago

Due date changed from 2017-10-11 to 2017-10-25

Actions

Copy link

#10

Updated by okurz over 6 years ago

Target version set to Milestone 11

Actions

Copy link

#11

Updated by okurz over 6 years ago

Assignee deleted (~~mgriessmeier~~)

mgriessmeier in vacation, unassigning for now.

Actions

Copy link

#12

Updated by zluo over 6 years ago

Assignee set to zluo

will try to fix this issue for now.

Actions

Copy link

#13

Updated by zluo over 6 years ago

shutdown reached target but it stops for further tests:

http://e13.suse.de/tests/4411

fixed however issue in shutdown.pm for textmode:

# s390x on SLE15 does not have a X11/VNC server
if (is_sle && sle_version_at_least('15') && check_var('ARCH', 's390x')) {
power_action('poweroff', textmode => 1);
}
power_action('poweroff');

Actions

Copy link

#14

Updated by riafarov over 6 years ago

Related to action #18936: [tools][sles][functional] Enable 3 stress acceptance on s390x added

Actions

Copy link

#15

Updated by riafarov over 6 years ago

Related to action #13216: [sles][functional][s390x] Run extratest on s390x added

Actions

Copy link

#16

Updated by zluo over 6 years ago

the problem is that after shutdown the worker still keeps X11 session alive. We need to find a way to terminate it.
Will discuss next week with @okurz and others.

Actions

Copy link

#17

Updated by zluo over 6 years ago

blocked at moment because of no idea to handle with running x11 session and s390-kvm is not available now...

Actions

Copy link

#18

Updated by okurz over 6 years ago

Status changed from New to In Progress

the ticket is not "new" anymore -> "in progress". Please discuss with mgriessmeier how to make sure we do not conflict each other with instances.

Actions

Copy link

#19

Updated by zluo over 6 years ago

atm I cannot work on this ticket because s390x-kvm is not ready.
I got an idea to workaround this issue with running x11 session:
create a needle and select_console and return

Actions

Copy link

#20

Updated by mgriessmeier over 6 years ago

Subject changed from [sles][functional][s390x] test fails in shutdown: VNC stall detected, needs to be investigated to [sles][functional][s390x][s390x-kvm] test fails in shutdown: VNC stall detected, needs to be investigated
Status changed from In Progress to Rejected
Assignee changed from zluo to mgriessmeier

We don't know if we ever saw this on production - linked job urls are a different problem

Actions

Copy link

#21

Updated by riafarov over 6 years ago

Status changed from Rejected to New

How failed in production on x-kvm. See https://openqa.suse.de/tests/1223349#

Actions

Copy link

#22

Updated by okurz over 6 years ago

Priority changed from Normal to Urgent

Fails in many more jobs in build 305.1

Actions

Copy link

#23

Updated by mgriessmeier over 6 years ago

Status changed from New to In Progress

Actions

Copy link

#24

Updated by mgriessmeier over 6 years ago

Subject changed from [sles][functional][s390x][s390x-kvm] test fails in shutdown: VNC stall detected, needs to be investigated to [sles][functional][s390x] test fails in shutdown: VNC stall detected, needs to be investigated
Status changed from In Progress to New

Actions

Copy link

#25

Updated by mgriessmeier over 6 years ago

Status changed from New to In Progress

Actions

Copy link

#26

Updated by mgriessmeier over 6 years ago

Was investigating this with riafarov.

We came to the conclusion that it's a weird backend behaviour in assert_shutdown in backend/testapi.pm
and therefore created a ticket for the tools team as a blocker for this: https://progress.opensuse.org/issues/26886

07:51:42.7571 4947 <<< testapi::type_string(string='poweroff
', max_interval=250, wait_screen_changes=0, wait_still_screen=0)
[[0;32m  OK  [0m] Stopped target Timers.
[[0;32m  OK  [0m] Stopped Daily Cleanup of Temporary Directories.
[[0;32m  OK  [0m] Stopped Early Kernel Boot Messages.
[[0;32m  OK  [0m] Stopped target Multi-User System.
         Stopping OpenSSH Daemon...
[[0;32m  OK  [0m] Stopped target Network is Online.
         Stopping Command Scheduler...
         Stopping Session 1 of user root.
[[0;32m  OK  [0m] Removed slice system-systemd\x2dhibernate\x2dresume.slice.
         Stopping Load kdump kernel and initrd...
         Stopping User Manager for UID 0...
07:51:42.9778 Debug: /var/lib/openqa/cache/tests/sle/tests/shutdown/shutdown.pm:32 called utils::power_action
[[0;32m  OK  [0m] Removed slice system-getty.slice.
07:51:42.9780 4947 <<< testapi::assert_shutdown(timeout=60)
[[0;32m  OK  [0m] Stopped /etc/init.d/after.local Compatibility.
07:51:43.0602 4949 Connection to root@s390p8.suse.de established
07:51:43.1630 4949 Command executed: ! virsh dominfo openQA-SUT-2 | grep -w 'shut off', ret=0
[[0;32m  OK  [0m] Stopped target Login Prompts.
[[0;32m  OK  [0m] Stopped Discard unused blocks once a week.
         Stopping System Logging Service...
         Stopping Restore /run/initramfs on shutdown...
         Stopping Load kdump kernel early on startup...
         Stopping Serial Getty on ttysclp0...
[[0;32m  OK  [0m] Stopped System Logging Service.
[[0;32m  OK  [0m] Stopped Serial Getty on ttysclp0.
[[0;32m  OK  [0m] Stopped OpenSSH Daemon.
[[0;32m  OK  [0m] Stopped Command Scheduler.
         Stopping Postfix Mail Transport Agent...
[[0;32m  OK  [0m] Stopped /etc/init.d/boot.local Compatibility.
[[0;32m  OK  [0m] Removed slice system-serial\x2dgetty.slice.
[[0;32m  OK  [0m] Stopped Session 1 of user root.
[[0;32m  OK  [0m] Stopped Restore /run/initramfs on shutdown.
[[0;32m  OK  [0m] Stopped Load kdump kernel early on startup.
[[0;32m  OK  [0m] Stopped Load kdump kernel and initrd.
[[0;32m  OK  [0m] Stopped Postfix Mail Transport Agent.
[[0;32m  OK  [0m] Stopped target Host and Network Name Lookups.
[[0;32m  OK  [0m] Stopped User Manager for UID 0.
[[0;32m  OK  [0m] Removed slice User Slice of root.
         Stopping Login Service...
         Stopping Permit User Sessions...
[[0;32m  OK  [0m] Stopped Login Service.
[[0;32m  OK  [0m] Stopped Permit User Sessions.
[[0;32m  OK  [0m] Stopped target User and Group Name Lookups.
         Stopping Name Service Cache Daemon...
[[0;32m  OK  [0m] Stopped target Network.
         Stopping wicked managed network interfaces...
[[0;32m  OK  [0m] Stopped target Remote File Systems.
[[0;32m  OK  [0m] Stopped target Remote File Systems (Pre).
[[0;32m  OK  [0m] Stopped Name Service Cache Daemon.
[[0;32m  OK  [0m] Stopped wicked managed network interfaces.
         Stopping wicked network nanny service...
[[0;32m  OK  [0m] Stopped wicked network nanny service.
         Stopping wicked network management service daemon...
[[0;32m  OK  [0m] Stopped wicked network management service daemon.
         Stopping wicked DHCPv4 supplicant service...
         Stopping wicked AutoIPv4 supplicant service...
         Stopping wicked DHCPv6 supplicant service...
[[0;32m  OK  [0m] Stopped wicked DHCPv4 supplicant service.
[[0;32m  OK  [0m] Stopped wicked DHCPv6 supplicant service.
[[0;32m  OK  [0m] Stopped wicked AutoIPv4 supplicant service.
         Stopping D-Bus System Message Bus...
[[0;32m  OK  [0m] Stopped D-Bus System Message Bus.
[[0;32m  OK  [0m] Stopped target Basic System.
[[0;32m  OK  [0m] Stopped target Sockets.
[[0;32m  OK  [0m] Closed Syslog Socket.
[[0;32m  OK  [0m] Stopped target Slices.
[[0;32m  OK  [0m] Removed slice User and Session Slice.
[[0;32m  OK  [0m] Stopped target Paths.
[[0;32m  OK  [0m] Closed D-Bus System Message Bus Socket.
[[0;32m  OK  [0m] Stopped target System Initialization.
[[0;32m  OK  [0m] Stopped Update is Completed.
[[0;32m  OK  [0m] Stopped target Swap.
         Deactivating swap /dev/disk/by-uuid…7d3-20c2-47b0-a965-339f4851c325...
[[0;32m  OK  [0m] Stopped target Encrypted Volumes.
[[0;32m  OK  [0m] Stopped Dispatch Password Requests to Console Directory Watch.
[[0;32m  OK  [0m] Stopped Rebuild Journal Catalog.
[[0;32m  OK  [0m] Stopped Apply Kernel Variables.
         Stopping Load/Save Random Seed...
[[0;32m  OK  [0m] Stopped Rebuild Hardware Database.
[[0;32m  OK  [0m] Stopped Commit a transient machine-id on disk.
         Stopping Update UTMP about System Boot/Shutdown...
[[0;32m  OK  [0m] Stopped Load Kernel Modules.
[[0;32m  OK  [0m] Stopped Update UTMP about System Boot/Shutdown.
[[0;32m  OK  [0m] Stopped Create Volatile Files and Directories.
[[0;32m  OK  [0m] Stopped Flush Journal to Persistent Storage.
[[0;32m  OK  [0m] Stopped target Local File Systems.
         Unmounting /var/lib/mariadb...
         Unmounting /var/lib/pgsql...
         Unmounting /var/crash...
         Unmounting /boot/grub2/s390x-emu...
         Unmounting /opt...
         Unmounting /var/lib/libvirt/images...
         Unmounting /var/cache...
         Unmounting /var/lib/named...
         Unmounting /run/user/0...
         Unmounting /.snapshots...
         Unmounting /boot/zipl...
         Unmounting /usr/local...
         Unmounting /var/opt...
         Unmounting /var/tmp...
         Unmounting /srv...
         Unmounting /var/lib/mysql...
         Unmounting /tmp...
         Unmounting /var/lib/machines...
         Unmounting /var/spool...
         Unmounting /var/lib/mailman...
         Unmounting /var/log...
[[0;32m  OK  [0m] Stopped Load/Save Random Seed.
[[0;32m  OK  [0m] Unmounted /usr/local.
[[0;32m  OK  [0m] Unmounted /var/tmp.
[[0;32m  OK  [0m] Unmounted /tmp.
[[0;32m  OK  [0m] Unmounted /var/lib/mailman.
[[0;32m  OK  [0m] Deactivated swap /dev/disk/by-path/ccw-0.0.0000-part3.
[[0;32m  OK  [0m] Deactivated swap /dev/disk/by-partu…6a711-92d2-4a69-8bab-84e4628c909e.
[[0;32m  OK  [0m] Deactivated swap /dev/vda3.
[[0;32m  OK  [0m] Deactivated swap /dev/disk/by-uuid/…027d3-20c2-47b0-a965-339f4851c325.
[[0;32m  OK  [0m] Unmounted /var/log.
[[0;32m  OK  [0m] Unmounted /.snapshots.
[[0;32m  OK  [0m] Unmounted /var/cache.
[[0;32m  OK  [0m] Unmounted /var/spool.
[[0;32m  OK  [0m] Unmounted /var/lib/pgsql.
[[0;32m  OK  [0m] Unmounted /var/crash.
[[0;32m  OK  [0m] Unmounted /boot/grub2/s390x-emu.
[[0;32m  OK  [0m] Unmounted /opt.
[[0;32m  OK  [0m] Unmounted /var/lib/libvirt/images.
[[0;32m  OK  [0m] Unmounted /run/user/0.
[[0;32m  OK  [0m] Unmounted /var/opt.
[[0;32m  OK  [0m] Unmounted /srv.
[[0;32m  OK  [0m] Unmounted /var/lib/machines.
[[0;32m  OK  [0m] Unmounted /var/lib/named.
[[0;32m  OK  [0m] Unmounted /var/lib/mysql.
[[0;32m  OK  [0m] Unmounted /var/lib/mariadb.
[[0;32m  OK  [0m] Unmounted /boot/zipl.
[[0;32m  OK  [0m] Reached target Unmount All Filesystems.
[[0;32m  OK  [0m] Stopped target Local File Systems (Pre).
[[0;32m  OK  [0m] Stopped Create Static Device Nodes in /dev.
[[0;32m  OK  [0m] Stopped Create System Users.
[[0;32m  OK  [0m] Stopped Remount Root and Kernel File Systems.
[[0;32m  OK  [0m] Reached target Shutdown.

Actions

Copy link

#27

Updated by mgriessmeier over 6 years ago

Blocked by action #26886: [tools][s390x-kvm] investigate and improve 'assert_shutdown' function in testapi added

Actions

Copy link

#28

Updated by mgriessmeier over 6 years ago

Description updated (diff)

Actions

Copy link

#29

Updated by okurz over 6 years ago

Description updated (diff)

We should keep one thing in mind. Unless I am mistaken we only have successfully executed "shutdown" on zkvm. So we shouldn't hunt for a regression on s390x-kvm and z/VM.
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP3&build=0473&arch=s390x are all SLE 12 SP3 GM s390x tests for reference. IIUC for s390x zVM we do not have an implementation for "is_shutdown" so there should be the message "Backend does not implement is_shutdown - just sleeping". For s390x-kvm that should be svirt calling ! virsh dominfo $vmname | grep -w 'shut off' which we see in autoinst-log.txt

I see an easy way out: We just skip everything that does not work on s390x (s390x-kvm and z/VM).

I tried to reproduce the problem on zVM locally with a simplified test plan by trying to shutdown but failed to show the problem. It just works fine: http://lord.arch/tests/7741/file/autoinst-log.txt

We have seen the problem only on s390x-kvm and now on zVM as well but not on zkvm, correct?

Actions

Copy link

#30

Updated by okurz over 6 years ago

Description updated (diff)

Actions

Copy link

#31

Updated by mgriessmeier over 6 years ago

for now we just skip assert_shutdown on s390x-kvm and z/VM:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3752

Actions

Copy link

#32

Updated by okurz over 6 years ago

Copied to action #26914: [sle][functional][s390x][s390x-kvm] s390x-kvm does never exit from "assert_shutdown" but zkvm works -> investigate how the machines differ, maybe problem on s390p8? added

Actions

Copy link

#33

Updated by okurz over 6 years ago

Status changed from In Progress to Feedback

mgriessmeier and me opted for the "easy way out" -> https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3752 , merged.

When jobs don't incomplete anymore we should create another ticket to do a more deep investigation why s390x-kvm and zkvm behave different here. I think the z/VM part is covered by #26886 because the backend implementation is basically a sleep only so it has nothing to do with actual execution on z/VM.

Rest handled in #26914

skip_registration s390x-kvm passed shutdown now

Can't close ticket? set to feedback now, something blocking here?

Actions

Copy link

#34

Updated by okurz over 6 years ago

Blocked by deleted (action #26886: [tools][s390x-kvm] investigate and improve 'assert_shutdown' function in testapi)

Actions

Copy link

#35

Updated by okurz over 6 years ago

Related to action #26886: [tools][s390x-kvm] investigate and improve 'assert_shutdown' function in testapi added

Actions

Copy link

#36

Updated by okurz over 6 years ago

Status changed from Feedback to Resolved

Not blocked by #26886 anymore, closing.

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

QA » openQA Project » openQA Tests

Tags

Custom queries

action #25638

[sles][functional][s390x] test fails in shutdown: VNC stall detected, needs to be investigated

Observation¶

Reproducible¶

Steps to reproduce¶

Expected result¶

Further details¶

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by riafarov over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by riafarov over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by zluo over 6 years ago

Updated by zluo over 6 years ago

Updated by riafarov over 6 years ago

Updated by riafarov over 6 years ago

Updated by zluo over 6 years ago

Updated by zluo over 6 years ago

Updated by okurz over 6 years ago

Updated by zluo over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by riafarov over 6 years ago

Updated by okurz over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by mgriessmeier over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago

Updated by okurz over 6 years ago