QA » openQA Project

https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-vcenter-server-70-release-notes.html

• Removal of VNC Server from ESXi

In vSphere 7.0, the ESXi built-in VNC server has been removed. Users will no longer be able to connect to a virtual machine using a VNC client by setting the RemoteDisplay.vnc.enable configure to be true. Instead, users should use the VM Console via the vSphere Client, the ESXi Host Client, or the VMware Remote Console, to connect virtual machines. Customers desiring VNC access to a VM should use the VirtualMachine.AcquireTicket("webmks") API, which offers a VNC-over-websocket connection. The webmks ticket offers authenticated access to the virtual machine console. For more information, please refer to the VMware HTML Console SDK Documentation(http://www.vmware.com/support/developer/html-console/).

okurz wrote:

@xlai as you moved the ticket into the openQA project: What is your intention, who do you suggest to work on this?

@okurz, thanks for the attention on the ticket and reply.

It was in our project because in the past backend issues were mainly excluded from tools' team scope, and this issue was blocking us from automating 7.0 tests, so we had to create one ticket for it.
To be honest, we do not have available skillful people to work on it, because after Michal Nowak left, we do not yet have anyone with time to get familiar with svirt backend code, and Nan(replacement of Michal) is just familiar with openqa test code now via implementing a kind of new open-vm-tools test and he will be busy with kubevirt automation test later. So we kind of stuck here.

With the discussion results last Wednesday and openqa tools team's great support on backend related tickets, I assume that this svirt backend extension to support connection to vmware 7.0(removed vnc), which is tightly related with backend connection methods and less related with testing logic(automation test), should show up in "openqa project", so I moved it here. I guess your team will gradually foster the skillful people to work on svirt backend tickets, right?

Do you already have a VMware 7.0 server running that shows that problems with openQA?

Yes, the issue was exposed when we automating vmware 7.0 open-vm-tools test. @nanzhang Can you help to provide a job link for the issue?

BTW, if your team does not have such available machine, we can setup one and provide you for development, but it will be in shared way because the machine will also be used for vmware testing. Or, you can provide HW and we can setup for you.

When you say "blocks all VT tests on 7.0", when would this apply?

This issue blocks guest installation on vmwrae 7.0, which is the fundamental test step, so blocking also further tests in our 15sp4 LTP (https://confluence.suse.com/display/qasleapac2/SLE15SP4+Virtualization-Function+Level+Test+Plan#SLE15SP4VirtualizationFunctionLevelTestPlan-VMware).

I suggest to prioritize this work below "High" as adding support for more recent VMware versions should be part of a properly planned extension work and not be seen as a regression, right?

Exactly.

Nan provided me the failure job on vmware vsphere 7.0, http://10.67.129.66/tests/299.

@okurz An update about vmware 7.0 test urgency for you to refer:

• vmware 7.0 becomes the only required vmware product to test for virtualization maintenance by VT project managers, this week
• vmware 7.0 will also become the only required vmware product to test for sle15sp5(6.5&6.7 out of vmware general support in Oct 2022).

It will be very helpful if this impediment can have a solution soon.

I would add the ticket to our backlog as soon as we can make it public. Can you explain what's the reason for the ticket being private? Do we need to keep this?

okurz wrote:

I would add the ticket to our backlog as soon as we can make it public. Can you explain what's the reason for the ticket being private? Do we need to keep this?

@okurz, thanks for the help! I rephrase the description, I think it should be fine to be public if this ticket is visible within SUSE only.

xlai wrote:

Do you already have a VMware 7.0 server running that shows that problems with openQA?

Yes, the issue was exposed when we automating vmware 7.0 open-vm-tools test. @nanzhang Can you help to provide a job link for the issue?

BTW, if your team does not have such available machine, we can setup one and provide you for development, but it will be in shared way because the machine will also be used for vmware testing. Or, you can provide HW and we can setup for you.

@xlai can you please provide access to a VMWare server where we could start looking into the feature development?

@xlai can you please provide access to a VMWare server where we could start looking into the feature development?

@okurz, We have one vmware 7.0 server which is used by official 15sp4 testing and located in BJ lab. We can lend it to you when we are not testing. But the required test environment is better to be not destroyed. Is it okay to share the machine with your development task?

xlai wrote:

@xlai can you please provide access to a VMWare server where we could start looking into the feature development?

@okurz, We have one vmware 7.0 server which is used by official 15sp4 testing and located in BJ lab. We can lend it to you when we are not testing. But the required test environment is better to be not destroyed. Is it okay to share the machine with your development task?

I am not sure what you mean but we can of course ask developers to not remove any VMs and be careful :)

okurz wrote:

I am not sure what you mean but we can of course ask developers to not remove any VMs and be careful :)

Great, thanks. We will do some HW preparation and provide you the hw access info via email tomorrow. It will be a vmware vsphere 7.0, which has web management interface. Let us know if any further needed.

You can also provide the hw access info here as private comment (so everyone in the team can see it and work on the ticket).

Thanks. I guess we can work with that.

okurz wrote:

Thanks. I guess we can work with that.

Great! Thank you and all tools team for the big support on this important featuer for us!

The login works but one cannot do much because the license is expired, see #106083#note-5. Otherwise you find my notes here: #106083#note-4

I also saw that the setup allows one to select compatibility for ESXi 6.x. I'm wondering whether using that is a possibility as well (at least until we can support 7.x).

@okurz, From your above comment#34, I can see that you really carefully evaluated where to go for this ticket, thanks for the efforts. I have been thinking about this since I saw the comment on Monday. I also discussed with Nan this morning. It is indeed a difficult decision to make for us, among the four choices:
a) implement in svirt backend to provide vm graphic interaction by VirtualMachine.AcquireTicket("webmks") API

From comment, your main concern of low ROI seems to come from that the test does not reflect real customers' usage. This is a valid point, however it is only tiny sacrifice. Graphic testing of vm has always been a low priority test for virtualization.

To be honest, I worry more that vmware stops support for VNC-over-websocket connection(webmks api) some day which results in low ROI. Otherwise it will be the best solution. I searched in vmware website and did not get clue about stopping this. However, they indeed encourage to use "VM remote console". See details in

• https://docs.vmware.com/en/VMware-Remote-Console/11.0/com.vmware.vmrc.vsphere.doc/GUID-703AA27D-1AF3-4067-BE5E-99C3D0032F38.html
• https://vdc-repo.vmware.com/vmwb-repository/dcr-public/ba6c113a-fb2a-4f68-8fe8-0930a4744013/4560a5a3-0cd6-40d8-a58a-70a53a0138de/vs600_vmrc_prog.pdf (api)

How do you think about implementing this ticket based on remote console sdk api in os-autoinst code? This will keep svirt capable to do graphic vm testing. Test code change will be low, but will need needle recreation?

b) Start VM with just serial terminal and spawn VNC server within the SUT, compare to s390x z/VM test implementations

QA impact evaluation: This solution can work -- support both command line tool and graphic test needs. Test code needs to be rewrite to some extent. No testing of graphic from virtualization support at all, only os level and above. From openqa tool regard, for svirt, no sut console to provide vm graphical interactivity test any more and less powerful to let testers stay on this backend.

c) Automate VMWare tooling as part of tests itself, e.g. the web interface

QA impact evaluation: This solution works too, but with most test code rewrite and needle recreation, which is not easy to afford.

d) Rely on only the serial terminal, similar as is done for public cloud tests maintained by the according squad

This does not work. We should keep graphic vm testing supported.

Besides, I see jeos testing on vmware requires same sut console to do graphic interaction with vm. So also invite jeos owners, @jlausuch and @mloviska , to share opinions on where to go for 7.0.

xlai wrote:

How do you think about implementing this ticket based on remote console sdk api in os-autoinst code?

Yes, as I stated we plan to still follow on with the specific tasks defined. However given the updated plan I expect the implementation to take potentially longer. In the best case we are done in some weeks, in worst case we will not be able to sufficiently support the direct VNC connection to VMWare instances as in before due to the changes in the VMWare stack itself.

I strongly recommend you look into alternatives like the proposed ones. Going with the "VNC in the SUT" approach could maybe even be less work than necessary changes to support the websocket based VNC which also mean necessary needle updates

okurz wrote:

xlai wrote:

How do you think about implementing this ticket based on remote console sdk api in os-autoinst code?

Yes, as I stated we plan to still follow on with the specific tasks defined. However given the updated plan I expect the implementation to take potentially longer. In the best case we are done in some weeks, in worst case we will not be able to sufficiently support the direct VNC connection to VMWare instances as in before due to the changes in the VMWare stack itself.

I strongly recommend you look into alternatives like the proposed ones. Going with the "VNC in the SUT" approach could maybe even be less work than necessary changes to support the websocket based VNC which also mean necessary needle updates

@okurz, Copy that. Let's see how things turn out with your teams' investigation goes further. We can discuss this again when a correct and more complete overview of all related stuff is got.

Personally, regardless of VT PO role, I really hope that we can find a solution that can keep svirt backend capable to provide the vm graphic interactivity for vmware7.0+. Otherwise, this backend won't be what it name tells and lose the attraction to openqa virt users who this backend aims for. From virt test needs , without this capability, svirt backend will nearly be fully replacable by any other backend that can simply provide ssh interactivity.

I know you must care about this too. So let's see, in reality, how much we can protect this in the solution making process. Of course, if after careful consideration, your team has to drop this capability, we can accept to select between "VNC in the SUT" approach and "Automate VMWare tooling as part of tests itself".

All, FYI. I am seeking for help from VT PM to contact vmware about their recommended tool which will be long term supported in their toolstack for doing vm interaction. Details in https://confluence.suse.com/pages/viewpage.action?pageId=945267415. Will keep you synced with result once I get.

xlai wrote:

All, FYI. I am seeking for help from VT PM to contact vmware about their recommended tool which will be long term supported in their toolstack for doing vm interaction. Details in https://confluence.suse.com/pages/viewpage.action?pageId=945267415. Will keep you synced with result once I get.

Hello all, this is the reply from virtualization project manager.

VMware VNC: after doing some research from my side too, i can confirmed the only way to get a VNC capabilities is to enable it in the webmks (web mouse keyboard screen). The latest version of the VMware HTML Console SDK is 2.1, oct2016, so we can consider this is stable, and they will not change the API in the futur.

@okurz @mkittler With this, I hope the worry about low ROI by potential frequent vmware tool stack change can be much lower now. Besides, 7.0 will be the only requested product to test for vmware in next product(sle15sp5). Shall this ticket be given high priority again?

xlai wrote:

xlai wrote:

All, FYI. I am seeking for help from VT PM to contact vmware about their recommended tool which will be long term supported in their toolstack for doing vm interaction. Details in https://confluence.suse.com/pages/viewpage.action?pageId=945267415. Will keep you synced with result once I get.

Hello all, this is the reply from virtualization project manager.

VMware VNC: after doing some research from my side too, i can confirmed the only way to get a VNC capabilities is to enable it in the webmks (web mouse keyboard screen). The latest version of the VMware HTML Console SDK is 2.1, oct2016, so we can consider this is stable, and they will not change the API in the futur.

@okurz @mkittler With this, I hope the worry about low ROI by potential frequent vmware tool stack change can be much lower now. Besides, 7.0 will be the only requested product to test for vmware in next product(sle15sp5). Shall this ticket be given high priority again?

Hi xlai, nice to see that confirmation. My understanding is that still the suggested and preferred ways for customers to connect to VMs is to use the VM-console in the vSphere Client or the ESXi Host Client or the VMware Remote Console. I don't see any change regarding the ROI and no change in our plans regarding what we can provide. I understand you would prefer to hear something different but I prefer to give honest and realistic estimates rather than false promises that we can't fulfill.

okurz wrote:

@okurz, Np, I'd like to talk realistically too :)

My understanding is that still the suggested and preferred ways for customers to connect to VMs is to use the VM-console in the vSphere Client or the ESXi Host Client or the VMware Remote Console. I don't see any change regarding the ROI

Would you please confirm if the low ROI conclusion comes mainly from above? I will discuss with VT Pjm. If he gives that testing sles guest via VM-console thru above 3 ways is neither important nor necessary regarding our test coverage in link (replace all vmware with 7.0 for sle15sp5), will it bring different conclusion on tools side?

no change in our plans regarding what we can provide. I understand you would prefer to hear something different but I prefer to give honest and realistic estimates rather than false promises that we can't fulfill.

Since the situation(guess/confirmation/test requirement) has been changing and may change again, to avoid misunderstanding, would you please explicitly state tools team's plan for low and high ROI separately?

@okurz, FYI. In the weekly QA&VT meeting(https://confluence.suse.com/pages/viewpage.action?pageId=972423852), I asked for virtualization project manager's input on whether it is important/necessary to test sles guest with vSphere Client/ESXi Host Client/VMware Remote Console from virtualization test perspective, the answer is NO. And he mentioned again that he read a lot of vmware documents and he knew that a lot of customers were using vnc connections, so he believed that the vnc connection would continue for long time.

Let me copy the minutes here:

It will be good if we can continue to use VNC connection for the test, PM/Prjm doesn't care about VMware's tools, which is not ours concern, which is not SUSE's test strategy, the requirement is whether SLES can work well on VMware as guest at all.

xlai wrote:

@okurz, FYI. In the weekly QA&VT meeting(https://confluence.suse.com/pages/viewpage.action?pageId=972423852), I asked for virtualization project manager's input on whether it is important/necessary to test sles guest with vSphere Client/ESXi Host Client/VMware Remote Console from virtualization test perspective, the answer is NO. And he mentioned again that he read a lot of vmware documents and he knew that a lot of customers were using vnc connections, so he believed that the vnc connection would continue for long time.

Let me copy the minutes here:

It will be good if we can continue to use VNC connection for the test, PM/Prjm doesn't care about VMware's tools, which is not ours concern, which is not SUSE's test strategy, the requirement is whether SLES can work well on VMware as guest at all.

ok, that is good to know. Thank you for crosschecking. So then I suggest we continue as planned. That is as of now two related tickets #107026 and #107032

okurz wrote:

All subtasks resolved. With this we see that we have complete VMWare 7.0 support in os-autoinst and therefore openQA. For sure we expect some issues to come up in the future based on when you and users will use this new support in tests.

Big thanks to tools team! We will start using this feature to enable vmware 7 virtualization test from this Augest. @nanzhang will still be the one in charge for this. Please keep tuned for issues if any.