Project

General

Profile

Actions

action #113201

closed

coordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation

coordination #100688: [epic][virtualization][3rd party hypervisor] Add svirt backend compatibility for vmware 7.0

Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console size:M

Added by mkittler over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-07-04
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Acceptance criteria

  • AC1: It is possible to connect to a VMWare virtual machine via VMWare's VNC interface (e.g. by specifying credentials via some test variables, possibly also requiring to use a special/new VNC-console-API from test code)
  • AC2: The connection is attempted to be restored if it breaks (which involves requesting the websocket URL from scratch and invoking dewebsockify again)
  • AC3: Documentation for AC1 exists, e.g. under os-autoinst/doc/backends.md
  • AC4: Behavior is tested against a real VMWare instance
  • AC5: Relevant code has unit tests

Suggestions

  • See commit message of https://github.com/os-autoinst/os-autoinst/pull/2091 and build your changes on top of that PR
  • See https://progress.opensuse.org/issues/100688#note-25 for VMWare credentials for running a test against the real VMWare instance
    • e.g. invoke the test like this (password redacted, use URL encoding when inserting the password): OS_AUTOINST_TEST_AGAINST_REAL_VMWARE_INSTANCE='https://root:…@vh002.qa2.suse.asia/2' OS_AUTOINST_DEWEBSOCKIFY_PORT=5900 prove -vI. t/27-consoles-vmware.t
  • Extend unit tests as suggested in the mentioned commit message

Out of scope

  • Deal with poor performance; there's likely not much we can do
  • Implement JPEG compression in our VNC module (it might help with performance but likely also cause problems with the pattern matching)
Actions #1

Updated by mkittler over 1 year ago

  • Description updated (diff)
Actions #2

Updated by mkittler over 1 year ago

  • Description updated (diff)
Actions #3

Updated by mkittler over 1 year ago

  • Description updated (diff)
Actions #4

Updated by okurz over 1 year ago

  • Priority changed from Normal to High
  • Target version set to Ready
Actions #5

Updated by okurz over 1 year ago

  • Subject changed from Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console to Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #6

Updated by okurz over 1 year ago

  • Tracker changed from coordination to action
Actions #7

Updated by mkittler over 1 year ago

  • Assignee set to mkittler
Actions #8

Updated by mkittler over 1 year ago

I've been trying to run a real svirt/VMWare test locally. Unfortunately I've set the wrong password. Seems like that were too many login attempts via SSH with the wrong password. At least I now cannot login anymore, not even via the web app on https://vh002.qa2.suse.asia (using credentials from https://progress.opensuse.org/issues/100688#note-25).

Could someone reset the password? Maybe something like faillock --user … --reset would help.

Actions #9

Updated by mkittler over 1 year ago

  • Status changed from Workable to In Progress

Draft to configure VNC over WebSockets usage via test distribution: https://github.com/Martchus/os-autoinst-distri-opensuse/pull/new/svirt-vmware-vnc-cfg
Maybe it makes more sense to configure this "under the hood" within backend code, though.

Actions #10

Updated by okurz over 1 year ago

mkittler wrote:

Draft to configure VNC over WebSockets usage via test distribution: https://github.com/Martchus/os-autoinst-distri-opensuse/pull/new/svirt-vmware-vnc-cfg
Maybe it makes more sense to configure this "under the hood" within backend code, though.

You can consider the test distribution the "rapid prototyping" platform for os-autoinst for now :)

Actions #11

Updated by mkittler over 1 year ago

Actions #12

Updated by openqa_review over 1 year ago

  • Due date set to 2022-07-26

Setting due date based on mean cycle time of SUSE QE Tools

Actions #13

Updated by mkittler over 1 year ago

I've now managed to test this with a real job but I've ran into

error: internal error: HTTP response code 500 for call to 'RegisterVM_Task'. Fault: ServerFaultCode - Current license or ESXi version prohibits execution of the requested operation.

so I've resorted to use https://openqaw8-vmware.qa.suse.de where I've got one step further:

[2022-07-12T12:32:04.055634+02:00] [info] ::: basetest::runtest: # Test died: VMWare web socket URL request failed: The object 'vim.VirtualMachine:10' has already been deleted or has not been completely created

So I looks like I need to implement a retry after errors like this because likely it is just the "or has not been completely created" part.


I will also extend the svrit backend documentation to explain how I set things up.

Actions #14

Updated by mkittler over 1 year ago

One step further, now the auth fails (despite correct credentials):

[2022-07-12T14:14:13.360057+02:00] [debug] VMWare web socket URL request failed: The object 'vim.VirtualMachine:10' has already been deleted or has not been completely created, trying 11 more times
[2022-07-12T14:14:22.379136+02:00] [debug] VMWare auth request failed: Cannot complete login due to an incorrect user name or password., trying 10 more times
[2022-07-12T14:14:32.397369+02:00] [debug] VMWare auth request failed: Cannot complete login due to an incorrect user name or password., trying 9 more times
Actions #15

Updated by mkittler over 1 year ago

Looks like the auth issue is because one is already logged in at this point and cannot login again, see VMWare's event log:

Benutzer kann sich nicht anmelden, weil er bereits angemeldet ist   Dienstag, 12. Juli 2022, 16:41:53 +0200 Info    Keine
Benutzer root@10.163.28.162 als Mojolicious (Perl) angemeldet   Dienstag, 12. Juli 2022, 16:41:48 +0200 Info    Keine

So I'm either clever about reusing a session or just change the user agent (one can apparently login multiple times with different user agents).


Clearing the cookies is sufficient to get past the auth error. And The object 'vim.VirtualMachine:10' has already been deleted was because the VM-ID was wrong. However, now I'm maybe as far as it gets with https://openqaw8-vmware.qa.suse.de because creating a VNC over WebSockets "ticket" like on https://vh002.qa2.suse.asia (which already has the newer version) doesn't work.

Actions #16

Updated by okurz over 1 year ago

At any time feel welcome to ask users to test themselves

Actions #17

Updated by mkittler over 1 year ago

Actions #18

Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

The PR https://github.com/os-autoinst/os-autoinst/pull/2091 should now be ready to be merged. The diff coverage is slightly below our target but it should be ok for now as the most important parts are covered (and some covered lines are just not tracked). Before we invest any more effort into improving this it would make sense to merge the PR to get more user feedback.

Actions #19

Updated by mkittler over 1 year ago

Now the changes to use VNC over WebSockets have been merged. It should fulfill all acceptance criteria. To use it, one still needs to set the test variable VMWARE_VNC_OVER_WS=1 (and possibly VMWARE_VNC_OVER_WS_INSECURE=1). I've mentioned that in #eng-testing and will hopefully get some feedback.

Actions #20

Updated by okurz over 1 year ago

  • Due date deleted (2022-07-26)
  • Status changed from Feedback to Resolved

With the above I agree that all ACs have been fulfilled. mkittler also mentioned and explained the current state in chat to important stakeholders so we can consider this done.

Actions

Also available in: Atom PDF