Project

General

Profile

Actions
Copy link

action #131189

closed

[qe-core] Introduce firewalld container test in ALP

Added by dvenkatachala 10 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
amanzini
Category:
New test
Target version:
QA - QE-Core: Ready
Start date:
2023-06-21
Due date:
% Done:

40%

Estimated time:
Difficulty:
Sprint:
QE-Core: November Sprint 23 (Nov 15 - Dec 13)
Tags:
qe-core-november-sprint

Description

We have the firewalld container in an ALP. We want to add a small test to test the firewalld container in the ALP. We need to set up multi-machine tests for this case.

https://github.com/thkukuk/firewalld-container#manage-firewalld-instance

Acceptance Criteria

  • AC1: PO to provide a base multimachine test to serve as inspiration/base to create the MM test on alp
  • AC2: Firewalld container workload is scheduled in Staging and Functional, on an encrypted scenario with Selinux enabled (which is the default)

Notes

Files

Screenshot from 2023-11-23 08-37-23.png (122 KB) Screenshot from 2023-11-23 08-37-23.png client-server networking with TAP and static network configured amanzini, 2023-11-23 07:41

Related issues 2 (0 open — 2 closed)

Related to openQA Tests - action #117616: [qe-core] How to start the firewalld containerResolveddvenkatachala2023-05-252023-05-25

Actions
Blocked by openQA Tests - action #135200: [qe-core] Implement a ping check with custom MTU packet sizeRejecteddvenkatachala2023-08-15

Actions
Actions
Copy link
 #1

Updated by dvenkatachala 10 months ago

  • Related to action #117616: [qe-core] How to start the firewalld container added
Actions
Copy link
 #2

Updated by dvenkatachala 10 months ago

  • Parent task set to #110092
Actions
Copy link
 #3

Updated by dvenkatachala 10 months ago

  • Subject changed from [qe-core] Introduce firewalld container test to [qe-core] Introduce firewalld container test in ALP
Actions
Copy link
 #4

Updated by szarate 8 months ago

  • Sprint set to QE-Core: September Sprint 23 (Sep 06 - Oct 04)
  • Tags set to qe-core-september-sprint
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by szarate 8 months ago

  • Blocked by action #135200: [qe-core] Implement a ping check with custom MTU packet size added
Actions
Copy link
 #6

Updated by szarate 8 months ago

  • Sprint deleted (QE-Core: September Sprint 23 (Sep 06 - Oct 04))
Actions
Copy link
 #7

Updated by szarate 7 months ago

  • Description updated (diff)
Actions
Copy link
 #8

Updated by szarate 7 months ago

  • Sprint set to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Actions
Copy link
 #9

Updated by szarate 7 months ago

  • Sprint changed from QE-Core: October Sprint 23 (Oct 11 - Nov 08) to QE-Core: September Sprint 23 (Sep 06 - Oct 04)
Actions
Copy link
 #10

Updated by szarate 7 months ago

  • Sprint changed from QE-Core: September Sprint 23 (Sep 06 - Oct 04) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Actions
Copy link
 #11

Updated by amanzini 6 months ago

  • Assignee set to amanzini

According to https://openqa.suse.de/tests/overview?distri=alp&version=1.0&build=B.8.29&groupid=524 There's already a firewalld test scheduled in ALP staging, but it's not a multimachine one

Actions
Copy link
 #12

Updated by amanzini 6 months ago

  • Status changed from Workable to In Progress
Actions
Copy link
 #13

Updated by szarate 5 months ago

Following up on: https://suse.slack.com/archives/C02CANHLANP/p1700117433682399

@amanzini Seeing the multiple things that are needed, I would like to also have the documentation or an extension of updated, on how to create MM tests :)

Actions
Copy link
 #14

Updated by amanzini 5 months ago · Edited

jotting down some notes while I work on this; first of all I failed because I was creating locking primitives (mutex) on the "child" job.
The child job is the one that contains the reference var PARALLEL_WITH: to the parent, so to make it properly work, we need to manage the schedule
like this

  - client:
      testsuite: null
      settings:
        [...]
        PARALLEL_WITH: 'server'           
  - server:
      testsuite: null
      settings:

on the server test code module, we should create the locking objects (mutex, barriers and so on) that will be shared with the client/child.

Network side, MM jobs must be using the 'tap' network . This means adding WORKER_CLASS=qemu_x86_64,tap on the settings and NICTYPE: tap in both jobs.
Thanks to PARALLEL_WITH, OpenQA will start VM on the worker on the same network, and provide two tap nics on the same virtual private connection.
The private network does not reach DHCP, so it must be manually configured.

The 'default' ignition.qcow2 virtual disk contains a combustion script that ends with a curl conncheck.opensuse.org . This fails if the machine has no proper external network dns/routing connectivity, so we need to stub out that statement and create a custom ignition_no_network_check.qcow2, that will be placed in openQA directory /var/lib/openqa/factory/hdd/ to be consumed.

How to mount a qcow2 disk image

This is a quick guide to mounting a qcow2 disk images on your host server. This is useful to reset passwords,
edit files, or recover something without the virtual machine running.

Step 1 - Enable NBD on the Host

modprobe nbd max_part=8

Step 2 - Connect the QCOW2 as network block device

qemu-nbd --connect=/dev/nbd0 /var/lib/vz/images/100/vm-100-disk-1.qcow2

Step 3 - Find The Virtual Machine Partitions

fdisk /dev/nbd0 -l

Step 4 - Mount the partition from the VM

mount /dev/nbd0p1 /mnt/somepoint/

Step 5 - After you done, unmount and disconnect

umount /mnt/somepoint/
qemu-nbd --disconnect /dev/nbd0
rmmod nbd
Actions
Copy link
 #15

Updated by amanzini 5 months ago · Edited

  • % Done changed from 0 to 10

looks like multi-machine networking is not stable ? (see attached screenshot). Doing some statistical investigation to see if it depends on the worker the job is being scheduled to

$ cat poo131189_ALP_containerd.sh 
#!/bin/bash
FL="Dolomite-Default"
VER=1.0
BLD=7.95
ARK="x86_64"
# test devel
GID="487"

/usr/bin/openqa-cli api -X --osd post isos \
  _SKIP_POST_FAIL_HOOKS=0 _SKIP_CHAINED_DEPS=1 \
  _GROUP_ID="$GID" DISTRI=alp VERSION="$VER" FLAVOR="$FL" BUILD="$BLD" ARCH="$ARK" \
  YAML_SCHEDULE=schedule/alp/firewalld_container.yaml WORKER_CLASS=qemu_x86_64,tap \
  CASEDIR=https://github.com/ilmanzo/os-autoinst-distri-opensuse#poo131189_ALP_firewalld \
  NICTYPE=tap 

https://openqa.suse.de/tests/12874043              passed     "host": "worker37",
https://openqa.suse.de/tests/12874044              passed     "host": "worker37",

https://openqa.suse.de/tests/12874045              passed     "host": "worker40",
https://openqa.suse.de/tests/12874046              passed     "host": "worker39",

https://openqa.suse.de/tests/12874047              failed     "host": "worker36",
https://openqa.suse.de/tests/12874048              passed     "host": "worker30",

https://openqa.suse.de/tests/12874049              failed     "host": "worker36",
https://openqa.suse.de/tests/12874050              passed     "host": "worker40",

https://openqa.suse.de/tests/12874051              passed     "host": "worker37",
https://openqa.suse.de/tests/12874052              passed     "host": "worker39",

https://openqa.suse.de/tests/12874053              failed     "host": "worker29",
https://openqa.suse.de/tests/12874054              passed     "host": "worker35",

https://openqa.suse.de/tests/12874055              failed     "host": "worker35",
https://openqa.suse.de/tests/12874056              passed     "host": "worker39",

https://openqa.suse.de/tests/12874057              passed     "host": "worker29",
https://openqa.suse.de/tests/12874058              passed     "host": "worker29",

https://openqa.suse.de/tests/12874059     parallel_failed     "host": "worker40",
https://openqa.suse.de/tests/12874060              failed     "host": "worker36",

https://openqa.suse.de/tests/12874061              passed     "host": "worker30",
https://openqa.suse.de/tests/12874062              passed     "host": "worker39",

https://openqa.suse.de/tests/12874076              passed     "host": "worker29",
https://openqa.suse.de/tests/12874077              passed     "host": "worker39",

https://openqa.suse.de/tests/12874078              failed     "host": "worker36",
https://openqa.suse.de/tests/12874079              passed     "host": "worker39",

https://openqa.suse.de/tests/12874080              passed     "host": "worker29",
https://openqa.suse.de/tests/12874081              passed     "host": "worker29",

https://openqa.suse.de/tests/12874082              failed     "host": "worker39",
https://openqa.suse.de/tests/12874083              passed     "host": "worker36",

https://openqa.suse.de/tests/12874084              passed     "host": "worker40",
https://openqa.suse.de/tests/12874085              passed     "host": "worker40",

https://openqa.suse.de/tests/12874086              passed     "host": "worker40",
https://openqa.suse.de/tests/12874087              passed     "host": "worker37",

https://openqa.suse.de/tests/12874088              passed     "host": "worker38",
https://openqa.suse.de/tests/12874089              passed     "host": "worker37",

https://openqa.suse.de/tests/12874090              passed     "host": "worker38",
https://openqa.suse.de/tests/12874091              failed     "host": "worker40",

https://openqa.suse.de/tests/12874092              passed     "host": "worker39",
https://openqa.suse.de/tests/12874093              passed     "host": "worker40",

https://openqa.suse.de/tests/12874094              failed     "host": "worker36",
https://openqa.suse.de/tests/12874095              passed     "host": "worker30",

https://openqa.suse.de/tests/12874096              passed     "host": "worker40",
https://openqa.suse.de/tests/12874097              passed     "host": "worker37",

https://openqa.suse.de/tests/12874098              passed     "host": "worker36",
https://openqa.suse.de/tests/12874099              passed     "host": "worker35",

https://openqa.suse.de/tests/12874100              failed     "host": "worker35",
https://openqa.suse.de/tests/12874101              passed     "host": "worker39",

https://openqa.suse.de/tests/12874102              passed     "host": "worker39",
https://openqa.suse.de/tests/12874103              passed     "host": "worker29",

https://openqa.suse.de/tests/12874104              passed     "host": "worker30",
https://openqa.suse.de/tests/12874105              passed     "host": "worker39",

https://openqa.suse.de/tests/12874106              passed     "host": "worker40",
https://openqa.suse.de/tests/12874107              passed     "host": "worker29",

https://openqa.suse.de/tests/12874108              failed     "host": "worker29",
https://openqa.suse.de/tests/12874109              passed     "host": "worker36",

https://openqa.suse.de/tests/12874110              passed     "host": "worker38",
https://openqa.suse.de/tests/12874111              passed     "host": "worker38",

https://openqa.suse.de/tests/12874112              failed     "host": "worker30",
https://openqa.suse.de/tests/12874113              passed     "host": "worker36",

https://openqa.suse.de/tests/12874114              failed     "host": "worker30",
https://openqa.suse.de/tests/12874115              passed     "host": "worker36",

https://openqa.suse.de/tests/12874116              passed     "host": "worker40",
https://openqa.suse.de/tests/12874117              passed     "host": "worker30",

https://openqa.suse.de/tests/12874118              passed     "host": "worker36",
https://openqa.suse.de/tests/12874119              passed     "host": "worker35",

https://openqa.suse.de/tests/12874120              passed     "host": "worker40",
https://openqa.suse.de/tests/12874121              passed     "host": "worker29",

https://openqa.suse.de/tests/12874122              failed     "host": "worker30",
https://openqa.suse.de/tests/12874123              passed     "host": "worker35",

https://openqa.suse.de/tests/12874124     parallel_failed     "host": "worker39",
https://openqa.suse.de/tests/12874125              failed     "host": "worker37",

https://openqa.suse.de/tests/12874397              passed     "host": "worker39",
https://openqa.suse.de/tests/12874398              passed     "host": "worker29",

https://openqa.suse.de/tests/12874399              failed     "host": "worker35",
https://openqa.suse.de/tests/12874400              passed     "host": "worker37",

https://openqa.suse.de/tests/12874401              failed     "host": "worker36",
https://openqa.suse.de/tests/12874402              passed     "host": "worker40",

https://openqa.suse.de/tests/12874403              passed     "host": "worker37",
https://openqa.suse.de/tests/12874404              passed     "host": "worker38",

https://openqa.suse.de/tests/12874405              passed     "host": "worker40",
https://openqa.suse.de/tests/12874406              passed     "host": "worker40",

https://openqa.suse.de/tests/12874407              passed     "host": "worker37",
https://openqa.suse.de/tests/12874408              passed     "host": "worker39",

https://openqa.suse.de/tests/12874409              passed     "host": "worker40",
https://openqa.suse.de/tests/12874410              passed     "host": "worker40",

https://openqa.suse.de/tests/12874411              failed     "host": "worker37",
https://openqa.suse.de/tests/12874412              passed     "host": "worker36",

https://openqa.suse.de/tests/12874413              failed     "host": "worker40",
https://openqa.suse.de/tests/12874414              passed     "host": "worker35",

https://openqa.suse.de/tests/12874415              failed     "host": "worker35",
https://openqa.suse.de/tests/12874416              passed     "host": "worker39",

https://openqa.suse.de/tests/12874417              passed     "host": "worker29",
https://openqa.suse.de/tests/12874418              passed     "host": "worker39",

https://openqa.suse.de/tests/12874419              failed     "host": "worker39",
https://openqa.suse.de/tests/12874420              passed     "host": "worker36",

https://openqa.suse.de/tests/12874421              passed     "host": "worker29",
https://openqa.suse.de/tests/12874422              passed     "host": "worker30",

https://openqa.suse.de/tests/12874423              passed     "host": "worker29",
https://openqa.suse.de/tests/12874424              passed     "host": "worker40",

https://openqa.suse.de/tests/12874425              passed     "host": "worker38",
https://openqa.suse.de/tests/12874426              passed     "host": "worker38",

https://openqa.suse.de/tests/12874427              failed     "host": "worker30",
https://openqa.suse.de/tests/12874428              passed     "host": "worker36",

https://openqa.suse.de/tests/12874429              failed     "host": "worker30",
https://openqa.suse.de/tests/12874430              passed     "host": "worker36",

https://openqa.suse.de/tests/12874431              passed     "host": "worker29",
https://openqa.suse.de/tests/12874432              passed     "host": "worker30",

https://openqa.suse.de/tests/12874433              passed     "host": "worker30",
https://openqa.suse.de/tests/12874434              passed     "host": "worker40",

https://openqa.suse.de/tests/12874435              failed     "host": "worker40",
https://openqa.suse.de/tests/12874436              passed     "host": "worker35",

https://openqa.suse.de/tests/12874437              passed     "host": "worker35",
https://openqa.suse.de/tests/12874438              passed     "host": "worker36",

https://openqa.suse.de/tests/12874439              failed     "host": "worker35",
https://openqa.suse.de/tests/12874440              passed     "host": "worker37",

https://openqa.suse.de/tests/12874441              passed     "host": "worker39",
https://openqa.suse.de/tests/12874442              passed     "host": "worker38",

https://openqa.suse.de/tests/12874443              failed     "host": "worker29",
https://openqa.suse.de/tests/12874444              passed     "host": "worker36",

https://openqa.suse.de/tests/12874445              passed     "host": "worker40",
https://openqa.suse.de/tests/12874446              passed     "host": "worker38",


# get job details
for j in $(cat results.txt| awk '{print $1}' | cut -d '/' -f 5) ; do openqa-cli api --osd jobs/$j | jq | grep assigned_worker ; done  > workers.txt

# associate with worker host
for w in $(cat workers.txt| awk '{print $2}' | cut -d, -f1) ; do openqa-cli api --osd workers/$w | jq | grep \"host ;  done  > worker_hosts.txt
Actions
Copy link
 #17

Updated by amanzini 5 months ago

  • % Done changed from 10 to 20

networking seems more stable on the next run:

12881342  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12881342              passed
12881343  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12881343              passed
12881344  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12881344              passed
12881345  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12881345              passed
12881346  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12881346              passed
12881347  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12881347              passed
12881348  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12881348              passed
12881349  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12881349              passed
12881350  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12881350              passed
12881351  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12881351              passed
Actions
Copy link
 #18

Updated by amanzini 5 months ago · Edited

tests are in a good shape. The only failing one seems related to nmcli connectivity check 

12893489  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893489              failed
12893490  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893490     parallel_failed
12893491  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893491              passed
12893492  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893492              passed
12893493  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893493              passed
12893494  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893494              passed
12893495  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893495              passed
12893496  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893496              passed
12893497  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893497              passed
12893498  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893498              passed
12893499  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893499              passed
12893500  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893500              passed
12893501  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893501              passed
12893502  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893502              passed
12893503  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893503              passed
12893504  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893504              passed
12893505  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893505              passed
12893506  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893506              passed
12893507  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893507              passed
12893508  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893508              passed
12893509  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893509              passed
12893510  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893510              passed
12893511  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893511              passed
12893512  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893512              passed
12893513  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893513              passed
12893514  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893514              passed
12893515  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893515              passed
12893516  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893516              passed
12893517  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893517              passed
12893518  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893518              passed
12893519  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893519              passed
12893520  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893520              passed
12893521  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893521              passed
12893522  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893522              passed
12893523  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893523              passed
12893524  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893524              passed
12893525  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893525              passed
12893526  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893526              passed
12893527  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893527              passed
12893528  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893528              passed
12893529  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893529              passed
12893530  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893530              passed
12893531  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893531              passed
12893532  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893532              passed
12893533  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893533              passed
12893534  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893534              passed
12893535  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893535              passed
12893536  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893536              passed
12893537  alp_firewalld_server-amanzini@64bit                        https://openqa.suse.de/tests/12893537              passed
12893538  alp_firewalld_client-amanzini@64bit                        https://openqa.suse.de/tests/12893538              passed
Actions
Copy link
 #19

Updated by amanzini 5 months ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #20

Updated by szarate 5 months ago

  • Sprint changed from QE-Core: October Sprint 23 (Oct 11 - Nov 08) to QE-Core: November Sprint 23 (Nov 15 - Dec 13)
Actions
Copy link
 #21

Updated by amanzini 5 months ago

  • Status changed from Feedback to In Progress
  • % Done changed from 20 to 30

PR merged , working on schedule

Actions
Copy link
 #22

Updated by amanzini 5 months ago · Edited

  • Status changed from In Progress to Feedback
  • % Done changed from 30 to 40

MR for functional schedule:
https://gitlab.suse.de/qe-core/qa-sle-functional-userspace/-/merge_requests/186

do we need to schedule it on all staging steps ? Or only on some/none ?

Actions
Copy link
 #23

Updated by amanzini 5 months ago

  • Status changed from Feedback to Resolved

test is green with new builds, so I'd consider this resolved.

https://openqa.suse.de/tests/12962353

Actions
Copy link
 #24

Updated by szarate 5 months ago · Edited

Q: how to schedule a job that uses a sle support server (already created one)?
Q: how to create a cluster with an alp support server

Jozef: In theory support server can be anything, it will just boot e.g. SLES image

Actions
Copy link

Also available in: Atom PDF