Project

General

Profile

action #106452

[opensuse] Smoke testing of Leap 15.4 Public Cloud images

Added by lkocman 6 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
openSUSE-Team-at-SUSE
Category:
New test
Target version:
-
Start date:
2022-02-09
Due date:
2022-05-27
% Done:

0%

Estimated time:
Difficulty:

Description

Action item from https://etherpad.opensuse.org/p/ReleaseEngineering-20220209#L32
We have a volunteer who'd like to help on the Public Cloud side of openSUSE.

What we're looking for is to re-use some basic tests from (SUSE-internal-only links)

http://openqa.suse.de/group_overview/274 EC2
http://openqa.suse.de/group_overview/219 Azure
http://openqa.suse.de/group_overview/275 GCE

To begin with we should cover upload to cloud and ensure that images boot.

I'm setting a deadline to 27th March to match with the GM submission deadline https://en.opensuse.org/openSUSE:Roadmap

Leap 15.4 Appliance images can be found here https://download.opensuse.org/distribution/leap/15.4/appliances/
openSUSE Leap 15.4 test suite can be found here https://openqa.opensuse.org/group_overview/50

I believe that openQA needles are stored here https://github.com/os-autoinst/os-autoinst-needles-opensuse/

We would be more than happy if we could reuse needles from SLE. There a help from Oliver or Santi is more than welcome.

History

#1 Updated by lkocman 6 months ago

  • Description updated (diff)

#2 Updated by lkocman 6 months ago

  • Description updated (diff)

#3 Updated by maritawerner 6 months ago

  • Tags set to qac

#4 Updated by jlausuch 6 months ago

The only concern I have about sharing current PC accounts for openSUSE testing is the exposure of the credentials, which are currently in a private repo in gitlab along with all the OSD workers. If the aim is to share the current accounts for SLE, we would need to involve our managers, as it might impact budget.

#5 Updated by zfeldstein 5 months ago

jlausuch wrote:

The only concern I have about sharing current PC accounts for openSUSE testing is the exposure of the credentials, which are currently in a private repo in gitlab along with all the OSD workers. If the aim is to share the current accounts for SLE, we would need to involve our managers, as it might impact budget.

Wondering if we could create instances in each cloud provider account and register them as openQA workers. We could leverage the cloud providers RBAC systems to assign our worker the ability to upload images/create instances. For example, in the case of AWS we could create an IAM role that has the requisite permissions and assign it to the worker instance. This would eliminate the need to expose the credentials, however this would increase the cloud spend in those accounts.

#6 Updated by szarate 4 months ago

  • Category set to New test

#7 Updated by jlausuch 4 months ago

  • Tags deleted (qac)

I would say, someone (the community) should come up with a proper planning for this, including budget plan.
Our budgets were planned to test SLE only (which was the original plan).

The second thing is to create a service principal for this purpose.

Third thing is to configure the credentials in the O3 workers and make sure those credentials are not exposed in plain text in openQA jobs (there are actually some effort invested in this).

Once that is sorted out, we are happy to share with you all the information and even having a workshop to explain how to use the Public Cloud libraries and tests that we have for SLE so that someone can also replicate that in O3.

#8 Updated by maritawerner 4 months ago

  • Subject changed from Smoke testing of Leap 15.4 Public Cloud images to [opensuse] Smoke testing of Leap 15.4 Public Cloud images
  • Assignee set to lkocman

lkocman could you please take care of that? I have no idea who else could sort that out.

#9 Updated by okurz 4 months ago

jlausuch wrote:

I would say, someone (the community) should come up with a proper planning for this, including budget plan.
Our budgets were planned to test SLE only (which was the original plan).

jlausuch I think being part of the community can be very beneficial and provide long-term sustainability. As a final product SLE might be of higher importance but please consider SUSE's Open Source policy https://www.suse.com/c/the-suse-open-source-policy/ , in particular "Factory First" which certainly should also apply for testing related work. In the best case you only need to provide the initial "catalyst" for the community to be able to further drive and support relevant work.

#10 Updated by jlausuch 4 months ago

okurz wrote:

jlausuch wrote:

I would say, someone (the community) should come up with a proper planning for this, including budget plan.
Our budgets were planned to test SLE only (which was the original plan).

jlausuch I think being part of the community can be very beneficial and provide long-term sustainability. As a final product SLE might be of higher importance but please consider SUSE's Open Source policy https://www.suse.com/c/the-suse-open-source-policy/ , in particular "Factory First" which certainly should also apply for testing related work. In the best case you only need to provide the initial "catalyst" for the community to be able to further drive and support relevant work.

I'm not sure why you are trying to teach me Factory First policy at this stage, probably you miss a lot of things we are doing for openSUSE, so please don't go this way.
And yes, I am willing to do provide any kind of help and support to the community as I'm doing every day of my work, but without budget plan, there is nothing to do, as Public Cloud testing is not free.

#11 Updated by zfeldstein 3 months ago

Who is a good contact to speak with about getting the appropriate resources configured/created in each respective cloud provider? If we already have Azure,GCE,AWS accounts, we could configure RBAC properly in each cloud provider for the instances to use and not expose credentials. If someone could provide access to the existing cloud accounts I could come up with a proposal and cost estimate for having either long lived openQA workers living in each provider, or "on demand" instances that are created when test are run.

#12 Updated by asmorodskyi 3 months ago

zfeldstein wrote:

Who is a good contact to speak with about getting the appropriate resources configured/created in each respective cloud provider? If we already have Azure,GCE,AWS accounts, we could configure RBAC properly in each cloud provider for the instances to use and not expose credentials. If someone could provide access to the existing cloud accounts I could come up with a proposal and cost estimate for having either long lived openQA workers living in each provider, or "on demand" instances that are created when test are run.

short answer current tests which we using to test Public Cloud instances in internal openQA instance is not possible to use in openqa.opensuse.org. You can schedule meeting with me and we will discuss details

#13 Updated by lkocman about 2 months ago

  • Assignee changed from lkocman to openSUSE-Team-at-SUSE

The budget question is resolved @Anton Smorodskyi controls the budget.

It's perfectly fine to use it in scope of Public Cloud testing for openSUSE.
Other usecases e.g. mirror in brazil or out of scope.

Lubos

#14 Updated by lkocman about 2 months ago

Jose could help with avoiding leaking the credentials.

#15 Updated by jlausuch about 2 months ago

lkocman wrote:

Jose could help with avoiding leaking the credentials.

lkocman: ok. We are in the process of setting up a VM with a service to provide the credentials via URL. This URL is protected with user/password which are secret variables in openQA, which means they are hidden and not visible in the logs, only reduced group of people with access to the worker machines (including myself) will be able to see them as they will be stored in /etc/openqa/workers.ini.

However, the VM that we have setup today is under suse.de and only available for OSD workers. We can do the same for O3, but we would need to have some Host or VM somewhere where O3 workers can reach.

#16 Updated by pdostal about 2 months ago

However, the VM that we have setup today is under suse.de and only available for OSD workers. We can do the same for O3, but we would need to have some Host or VM somewhere where O3 workers can reach.
Do we need a VM just to host a bunch of static files?

#17 Updated by jlausuch about 2 months ago

pdostal wrote:

However, the VM that we have setup today is under suse.de and only available for OSD workers. We can do the same for O3, but we would need to have some Host or VM somewhere where O3 workers can reach.
Do we need a VM just to host a bunch of static files?

They could reside in the same worker that we could use for PC tests.. however, we need to provide them as web service. Maybe a container on the worker should suffice? (thinking loud)

#18 Updated by dimstar about 2 months ago

jlausuch wrote:

They could reside in the same worker that we could use for PC tests.. however, we need to provide them as web service. Maybe a container on the worker should suffice? (thinking loud)

We have https://static.opensuse.org/ (https://github.com/openSUSE/static.opensuse.org) - if we don't plan on DoS'ing that host with gigabytes of transfers, that might be a good fit? (Not sure I got the requirements straight .. sounded like some static stuff being served)

#19 Updated by asmorodskyi about 2 months ago

dimstar wrote:

jlausuch wrote:

They could reside in the same worker that we could use for PC tests.. however, we need to provide them as web service. Maybe a container on the worker should suffice? (thinking loud)

We have https://static.opensuse.org/ (https://github.com/openSUSE/static.opensuse.org) - if we don't plan on DoS'ing that host with gigabytes of transfers, that might be a good fit? (Not sure I got the requirements straight .. sounded like some static stuff being served)

as we speaking here about very confidential data here I would not mix it with something which has totally different purpose on my understanding . Also again due to same reason I really prefer to keep it hidden in same network as openQA workers are

#20 Updated by asmorodskyi about 2 months ago

jlausuch wrote:

lkocman wrote:

Jose could help with avoiding leaking the credentials.

lkocman: ok. We are in the process of setting up a VM with a service to provide the credentials via URL. This URL is protected with user/password which are secret variables in openQA, which means they are hidden and not visible in the logs, only reduced group of people with access to the worker machines (including myself) will be able to see them as they will be stored in /etc/openqa/workers.ini.

However, the VM that we have setup today is under suse.de and only available for OSD workers. We can do the same for O3, but we would need to have some Host or VM somewhere where O3 workers can reach.

AFAIK O3 workers are running in isolated network , it would be perfect if we could install web server on any machine running in this subnet it could be apache/nginx or anything else which capable to expose directory structure via HTTPS and protect it with password.

#21 Updated by pdostal about 2 months ago

dimstar wrote:

jlausuch wrote:

They could reside in the same worker that we could use for PC tests.. however, we need to provide them as web service. Maybe a container on the worker should suffice? (thinking loud)

We have https://static.opensuse.org/ (https://github.com/openSUSE/static.opensuse.org) - if we don't plan on DoS'ing that host with gigabytes of transfers, that might be a good fit? (Not sure I got the requirements straight .. sounded like some static stuff being served)

as we speaking here about very confidential data here I would not mix it with something which has totally different purpose on my understanding . Also again due to same reason I really prefer to keep it hidden in same network as openQA workers are
+1

#22 Updated by jlausuch about 2 months ago

Let's just use openqaworker1 for this purpose. Podman is installed there and I we just need to add the _SECRET_ vars to /etc/openqa/workers.ini and run a container with the credentials web service...
The jobs will just need WORKER_CLASS=openqaworker1.

#23 Updated by asmorodskyi about 2 months ago

jlausuch wrote:

Let's just use openqaworker1 for this purpose. Podman is installed there and I we just need to add the _SECRET_ vars to /etc/openqa/workers.ini and run a container with the credentials web service...
The jobs will just need WORKER_CLASS=openqaworker1.

+1

#24 Updated by okurz about 2 months ago

asmorodskyi wrote:

AFAIK O3 workers are running in isolated network , it would be perfect if we could install web server on any machine running in this subnet it could be apache/nginx or anything else which capable to expose directory structure via HTTPS and protect it with password.

There is already a generic webserver within the o3 network, apache on o3 itself. And it serves the assets from /var/lib/openqa so just put something there which can even be downloaded from elsewhere with the help of openQA features and then where necessary adapt the openQA config.

A container on openqaworker1 is also fine but no permanent solution as we can't guarantee for any worker to stay up indefinitely.

#25 Updated by jlausuch about 2 months ago

okurz wrote:

There is already a generic webserver within the o3 network, apache on o3 itself. And it serves the assets from /var/lib/openqa so just put something there which can even be downloaded from elsewhere with the help of openQA features and then where necessary adapt the openQA config.

Is this webserver accessible from any public place, or only O3 workers can access to it?

#26 Updated by asmorodskyi about 2 months ago

May I suggest to define some sync call to find a common ground regarding all this ? I think it would be more effective way to find solution which suits everyone

#27 Updated by okurz about 2 months ago

jlausuch wrote:

okurz wrote:

There is already a generic webserver within the o3 network, apache on o3 itself. And it serves the assets from /var/lib/openqa so just put something there which can even be downloaded from elsewhere with the help of openQA features and then where necessary adapt the openQA config.

Is this webserver accessible from any public place, or only O3 workers can access to it?

The webserver is generally available from outside but parts are restricted to the internal network. Look into o3:/etc/apache2/vhosts.d/openqa.conf for some examples already doing that.

#28 Updated by jlausuch about 2 months ago

okurz wrote:

Is this webserver accessible from any public place, or only O3 workers can access to it?

The webserver is generally available from outside but parts are restricted to the internal network. Look into o3:/etc/apache2/vhosts.d/openqa.conf for some examples already doing that.

Ok, sounds interesting for our use case. Not sure if we still should isolate things in containers, but let's discuss it further.

asmorodskyi wrote:

May I suggest to define some sync call to find a common ground regarding all this ? I think it would be more effective way to find solution which suits everyone

+1
lkocman can you arrange some meeting to discuss this? Attendees: Anton, Oliver, myself and whoever you consider (maybe Pavel and Dimstar also wanna join?).

#29 Updated by lkocman about 2 months ago

Talk is scheduled for tomorrow at 14:30 CEST (teams).

Also available in: Atom PDF