action #168097: [qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de - openQA Tests (public) - openSUSE Project Management Tool

Actions

Copy link

action #168097

open

[qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de

Added by okurz 3 months ago. Updated about 6 hours ago.

Status:

Blocked

Priority:

Normal

Assignee:

dzedro

Category:

Enhancement to existing tests

Target version:

QA (public) - QE-Core: Ready

Start date:

2024-11-28

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Difficulty:

Sprint:

QE-Core: January Sprint 25 (Jan 09 - Feb 04)

Tags:

platform-team

Description

Motivation¶

Upcoming changes to the network will have the effect that dist.suse.de is not accessible anymore for any workload running in NUE2 hence osado tests relying on that should use https://mirror.nue2.suse.org/ instead of corresponding paths in http://dist.suse.de/ along with ensuring that relevant content is available already as https://mirror.nue2.suse.org/ might only have the relevant test assets available delayed.

Acceptance criteria¶

AC1: osado tests do not rely on http://download.suse.de when running from NUE2
AC2: osado tests do not rely on NUE2 resources when running from other locations
AC3.1: Mirrors are defined in workerconf.sls

Notes¶

globally default value (mirror.nue2) for all of the workers, for workers in CC area (zone-cc) https://download.suse.de.

Out of scope¶

AC3.2: qem-bot handles also different mirrors here

Suggestions¶

Review occurences of git grep dist.suse.de within osado and provide means to override the path based on openQA test variables
Same for download.suse.de
Same for https://github.com/openSUSE/qem-bot/ see here
Consider setting local mirror variables in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls

Subtasks 1 (0 open — 1 closed)

Related issues 4 (2 open — 2 closed)

Actions

Copy link

Updated by okurz 3 months ago

Parent task set to #166598

Actions

Copy link

Updated by okurz 3 months ago

Project changed from QA (public) to openQA Tests (public)
Subject changed from Make openqa.suse.de tests work with mirrors instead of dist.suse.de to [core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de
Description updated (diff)
Category set to Enhancement to existing tests
Assignee deleted (~~okurz~~)

Actions

Copy link

Updated by okurz 3 months ago

Target version changed from Ready to QE-Core: Ready

@szarate moving this to "QE-Core: Ready" to bring this ticket to your attention. Feel welcome to move it elsewhere.

Actions

Copy link

Updated by szarate 3 months ago

Tags set to platform-team
Subject changed from [core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de to [qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de

Actions

Copy link

Updated by szarate 3 months ago

Description updated (diff)

Actions

Copy link

#10

Updated by dzedro 3 months ago

Status changed from New to In Progress
Assignee set to dzedro

Actions

Copy link

#11

Updated by szarate 3 months ago · Edited

I take that http://dist.suse.de and http://download.suse.de are the same or contain the same information? is that assumption correct?

Also: https://confluence.suse.com/display/~moroni_flores/embargoed-proxy

Actions

Copy link

#12

Updated by dzedro 3 months ago · Edited

I wanted simply replace in medium types download.suse.de & dist.suse.de with mirror.nue2.suse.org, but looks like not everything works
This works
OS_TEST_REPOS=https://download.suse.de/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/
This does not work
OS_TEST_REPOS=https://mirror.nue2.suse.org/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/

or
works
QA_HEAD_REPO=http://dist.suse.de/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://dist.suse.de/ibs/QA:/SLE15SP2/update/
does not work
QA_HEAD_REPO=http://mirror.nue2.suse.org/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://mirror.nue2.suse.org/QA:/SLE15SP2/update/

Actions

Copy link

#13

Updated by dzedro 3 months ago

@okurz what does this mean ? AC3.1: Mirrors are defined in workerconf.sls

Actions

Copy link

#14

Updated by dzedro 3 months ago · Edited

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 workerconf.sls

Actions

Copy link

#15

Updated by mkittler 3 months ago

I've seen your qem-bot PR, see my comment on GitHub. I'll review the pillars change.

Note that I have been working on #168115 which had a similar goal. Checkout comment #168115#note-14 for my findings (or rather problems I ran into; I was not really successful in using the mirror for maintenance test).

Actions

Copy link

#16

Updated by dzedro 2 months ago · Edited

IDK what is the point of this changes when we will still use download/dist.suse.de
Also this and probably other repos should be mirrored on mirror.nue2.suse.org ?

Actions

Copy link

#17

Updated by szarate 2 months ago

@dzedro are those test settings?

if so, I would prefer to use variable expansion, QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.

Changes to qam-bot are clearly out of scope for qe-core.

Actions

Copy link

#18

Updated by szarate 2 months ago

Related to action #169159: Allow variable expansion incorporating worker settings size:S added

Actions

Copy link

#19

Updated by szarate 2 months ago

Description updated (diff)

Actions

Copy link

#20

Updated by szarate 2 months ago

Marking AC3.2 as out of scope; changes required exceed what qe-core can do.

Actions

Copy link

#21

Updated by dzedro 2 months ago

szarate wrote in #note-17:

@dzedro are those test settings?

if so, I would prefer to use variable expansion, QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.

Changes to qam-bot are clearly out of scope for qe-core.

I got this from db, this is mostly defined in medium types or in group job yamls
I agree REPO_MIRROR_HOST should be used, not sure where should this be defined on worker or jobs group or ?

Actions

Copy link

#22

Updated by szarate 2 months ago · Edited

Should be defined at worker level, https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 and on the test distribution, we need to die when it is not found (i.e check/get required var), or its empty.

Actions

Copy link

#23

Updated by waynechen55 2 months ago · Edited

Generally speaking, following resources need to be synced to https://mirror.nue2.suse.org/ for virtualization functional:

1 Installation media of SLES products. For versions have both Online and Full mediums, Full medium is required:

#SLES12
source.http.sles-12-sp3-64=http://schnell.nue.suse.com/SLE12/SLE-12-SP3-Server-GM/x86_64/DVD1/
source.http.sles-12-sp4-64=http://dist.suse.de/install/SLP/SLE-12-SP4-Server-GM/x86_64/DVD1/
source.http.sles-12-sp5-64=http://dist.suse.de/install/SLP/SLE-12-SP5-Server-GM/x86_64/DVD1/
#SLES15
source.http.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/
source.http.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/
source.http.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/
source.http.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/
source.http.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/
source.http.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/

#sle15 sp2 module repos x86_64
x86_64.source.Basesystem.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp3 module repos x86_64
x86_64.source.Basesystem.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp4 module repos x86_64
x86_64.source.Basesystem.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp5 module repos x86_64
x86_64.source.Basesystem.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp6 module repos x86_64
x86_64.source.Basesystem.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp7 module repos x86_64
x86_64.source.Basesystem.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Web-Scripting/

2 virt test related repos:

https://download.suse.de/ibs/Devel:/Virt:/Tests/

3 virt devel related repos:

https://download.suse.de/ibs/Devel:/Virt:/

4 virt update related repos:

http://download.suse.de/ibs/SUSE/Updates

5 virt update milestone repos:

http://download.suse.de/ibs/SUSE:/

Actions

Copy link

#24

Updated by szarate 2 months ago

@waynechen55 please create a subticket of https://progress.opensuse.org/issues/168448 with the information from your comment above, incl where are those defined (i.e where is source.http.sles-12-sp5-64 being defined, etc)

Actions

Copy link

#25

Updated by lmanfredi 2 months ago

For YaST , we are using the following resources that need to be synced

Actions

Copy link

#26

Updated by szarate 2 months ago · Edited

lmanfredi wrote in #note-25:

For YaST , we are using the following resources that need to be synced

http://dist.suse.de/ibs/SUSE:/Maintenance:/Test:/SLE-INSTALLER:/

http://dist.suse.de/ibs/SUSE:/Maintenance:/Test:/SLE-SERVER-INSTALLER:/12-SP5:/

http://dist.suse.de/ibs/SUSE:/SLE-15:/Update/standard/x86_64/

@lmanfredi https://progress.opensuse.org/issues/168097#note-24

Actions

Copy link

#27

Updated by szarate 2 months ago

Description updated (diff)

Actions

Copy link

#28

Updated by szarate 2 months ago

Sprint set to QE-Core: November Sprint 24 (Nov 06 - Dec 04)

Actions

Copy link

#29

Updated by mkittler about 2 months ago · Edited

This also concerns baremetal tests like this secnario running on squiddlydiddly. Here the problem is more tricky because the iPXE boot script is using assets provided by openQA:

[2024-11-26T09:43:54.202432Z] [debug] [pid:63172] setting iPXE bootscript to:
  #!ipxe
  echo ++++++++++++++++++++++++++++++++++++++++++
  echo ++++++++++++ openQA ipxe boot ++++++++++++
  echo +    Host: squiddlydiddly.qe.nue2.suse.org
  echo ++++++++++++++++++++++++++++++++++++++++++

  kernel http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1/boot/aarch64/linux install=http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1   regurl=http://all-41.1.proxy.scc.suse.de  console=ttyAMA0,115200  root=/dev/ram0  initrd=initrd  textmode=1  ssh=1 sshpassword=nots3cr3t  vnc=1 VNCPassword=nots3cr3t  plymouth.enable=0  Y2DEBUG=1 linuxrc.log=/dev/ttyAMA0 linuxrc.core=/dev/ttyAMA0 linuxrc.debug=4,trace  reboot_timeout=0 
  initrd http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1/boot/aarch64/initrd
  boot

(Those are logs from https://openqa.suse.de/tests/16002467.)

EDIT: I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/941 to avoid running those jobs for now. This commit can be reverted when tests have been somehow adapted. Maybe it makes sense to create a separate ticket for that. We should probably coordinate with @MMoese.

Actions

Copy link

#30

Updated by MMoese about 2 months ago

Testing those assets from openQA is the whole point of this.. it doesn't matter if it is provided by openQA or any form of mirror, we can easily adopt that. But we need exactly these assets.

Actions

Copy link

#31

Updated by okurz about 2 months ago · Edited

Let me create another ticket for that different case

EDIT: Created #170362 to cover this

Actions

Copy link

#32

Updated by okurz about 2 months ago

Copied to action #170362: [qe-core] Make openqa.suse.de SLE product validation bare-metal tests work within NUE2 added

Actions

Copy link

#33

Updated by szarate about 2 months ago

Blocks action #170422: [qe-core] Jobs running in NUE cannot access required repository in openqa.suse.de - Valid metadata not found at specified URL - auto_review:"Timeout exceeded when accessing.*repomd.xml" added

Actions

Copy link

#34

Updated by szarate about 2 months ago

I cloned: https://openqa.suse.de/tests/16036071

openqa-clone-job --within-instance https://openqa.suse.de/tests/16032501 --skip-chained-deps QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% MIRROR_FTP=ftp://%REPO_MIRROR_HOST%/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTP=http://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTPS=https://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_NFS=nfs://%REPO_MIRROR_HOST%/var/lib/openqa/share/factory/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_SMB=smb://%REPO_MIRROR_HOST%/inst/SLE-15-SP7-Full-aarch64-Build43.1-Media1

And failed

due to some hardcoded stuff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20698

I just triggered: https://openqa.suse.de/tests/16036115

@dzedro if this works, we can move on to do the replacements

Actions

Copy link

#35

Updated by szarate about 2 months ago

Related to action #170491: [Containers] Some zkvm jobs can't to connect to via ftp download.suse.de for s390x added

Actions

Copy link

#36

Updated by szarate about 2 months ago

Related to deleted (action #170491: [Containers] Some zkvm jobs can't to connect to via ftp download.suse.de for s390x)

Actions

Copy link

#37

Updated by szarate about 2 months ago

Subtask #170491 added

Actions

Copy link

#38

Updated by okurz about 1 month ago

Related to action #169918: [security] tests never get executed on tyrion and coppi added

Actions

Copy link

#39

Updated by MMoese about 1 month ago

I don't see why we should have all baremetal machines with changed WORKER_CLASS, please change that back.

I don't see why we should not run those machines? We can boot, we can use mirrors, we have a working worker host. So please revert that change.

Actions

Copy link

#40

Updated by okurz about 1 month ago

Are you referring to #168097-29 ? Then I suggest you provide verification runs of what works and then one could adjust worker classes and test schedule accordingly

Actions

Copy link

#41

Updated by MMoese about 1 month ago · Edited

I just manually booted a machine (sonic) from osd and installed the latest build of sp7.

sonic-1:~ # ping openqa.suse.de
PING openqa.suse.de(openqa.oqa.prg2.suse.org (2a07:de40:b203:12:0:ff:fe4f:7c2b)) 56 data bytes
64 bytes from openqa.oqa.prg2.suse.org (2a07:de40:b203:12:0:ff:fe4f:7c2b): icmp_seq=1 ttl=62 time=4.94 ms

Actions

Copy link

#42

Updated by dzedro 26 days ago

Status changed from In Progress to Blocked

https://progress.opensuse.org/issues/169327 and other repos have to be migrated

Actions

Copy link

#44

Updated by okurz 26 days ago

Blocking on ##169327

Actions

Copy link

#45

Updated by openqa_review 12 days ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: fips_install_lvm_full_encrypt
https://openqa.suse.de/tests/16039583#step/bootloader_start/1

To prevent further reminder comments one of the following options should be followed:

The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
The openQA job group is moved to "Released" or "EOL" (End-of-Life)
The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Actions

Copy link

#46

Updated by mgrifalconi 6 days ago

Sprint changed from QE-Core: November Sprint 24 (Nov 06 - Dec 04) to QE-Core: January Sprint 25 (Jan 09 - Feb 04)

Actions

Copy link

#47

Updated by dzedro about 6 hours ago

I created PR to add back global variable REPO_MIRROR_HOST https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/958
It is required to make expanding of REPO_MIRROR_HOST work.
Revert use of REPO_MIRROR_HOST on s390x ftp repo https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20934
All s390x workers are in PRG and use openqa.suse.de

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

QA (public) » openQA Project (public) » openQA Tests (public)

Tags

Custom queries

action #168097

[qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de

Motivation¶

Acceptance criteria¶

Notes¶

Out of scope¶

Suggestions¶

Updated by okurz 3 months ago

Updated by okurz 3 months ago

Updated by okurz 3 months ago

Updated by szarate 3 months ago

Updated by szarate 3 months ago

Updated by dzedro 3 months ago

Updated by szarate 3 months ago · Edited

Updated by dzedro 3 months ago · Edited

Updated by dzedro 3 months ago

Updated by dzedro 3 months ago · Edited

Updated by mkittler 3 months ago

Updated by dzedro 2 months ago · Edited

Updated by szarate 2 months ago

Updated by szarate 2 months ago

Updated by szarate 2 months ago

Updated by szarate 2 months ago

Updated by dzedro 2 months ago

Updated by szarate 2 months ago · Edited

Updated by waynechen55 2 months ago · Edited

Updated by szarate 2 months ago

Updated by lmanfredi 2 months ago

Updated by szarate 2 months ago · Edited

Updated by szarate 2 months ago

Updated by szarate 2 months ago

Updated by mkittler about 2 months ago · Edited

Updated by MMoese about 2 months ago

Updated by okurz about 2 months ago · Edited

Updated by okurz about 2 months ago

Updated by szarate about 2 months ago

Updated by szarate about 2 months ago

Updated by szarate about 2 months ago

Updated by szarate about 2 months ago

Updated by szarate about 2 months ago

Updated by okurz about 1 month ago

Updated by MMoese about 1 month ago

Updated by okurz about 1 month ago

Updated by MMoese about 1 month ago · Edited

Updated by dzedro 26 days ago

Updated by okurz 26 days ago

Updated by openqa_review 12 days ago

Updated by mgrifalconi 6 days ago

Updated by dzedro about 6 hours ago