openQA Project (public) » openQA Tests (public) » Containers and images

PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17793/files

Some failures in s390. Two things to do:

• Fix the problem: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17881
• Extend the test coverage

There is also an issue on PC: https://openqa.suse.de/tests/12339051#step/podman_privileged_mode/41

• https://openqa.suse.de/tests/12339067#
• https://openqa.suse.de/tests/12339068#
• https://openqa.suse.de/tests/12339070#

...

Any idea what could we add to improve the test?

This PR tests podman in podman, and docker in podman

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17901

We had an issue in https://openqa.suse.de/tests/12372015# but because a issues during the download of the image (not because the test itself)
With this PR is more resilient

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17909

Temporarily disabled due to ongoing test issues on s390x and others: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17911

Failure report:

http://mgrossu-openqa.qe.suse.de/tests/205#step/podman_privileged_mode/57
http://mgrossu-openqa.qe.suse.de/tests/210#step/podman_privileged_mode/57

To solve the problems we detected

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17921

We still need to check why /dev/bus was not working on JeOS, see https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/17892#issuecomment-1746212974

Thank you Ivan! So, the still open points are:

• Increase the test coverage, so far we have only a small smoke test. e.g. loading of a kernel module would be nice.
• https://progress.opensuse.org/issues/135518#note-13

docker_privileged_mode is failing on 15-SP1 https://openqa.suse.de/tests/12564822#step/docker_privileged_mode/82

dzedro wrote in #note-16:

docker_privileged_mode is failing on 15-SP1 https://openqa.suse.de/tests/12564822#step/docker_privileged_mode/82

time="2023-10-19T14:01:58Z" level=info msg="Received shutdown.Stop(), terminating!" PID=1571
time="2023-10-19T14:01:58Z" level=debug msg="Made network namespace at /run/netns/netns-b3666979-98c8-d000-ed83-a27eb93dd5e4 for container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144"
time="2023-10-19T14:01:58Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/merged: invalid argument"
time="2023-10-19T14:01:58Z" level=debug msg="Failed to mount container \"d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144\": creating overlay mount to /var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/merged, mount_data=\"lowerdir=/var/lib/containers/storage/overlay/l/OZK4FTOWYDDQBVKGJFHT7OWIZL,upperdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/diff,workdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/work,nodev\": using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first\nfuse-overlayfs: cannot mount: No such file or directory\n: exit status 1"
time="2023-10-19T14:01:58Z" level=debug msg="cni result for container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144 network podman: &{0.4.0 [{Name:cni-podman0 Mac:da:a6:11:74:4d:54 Sandbox:} {Name:vethbe33e7a9 Mac:12:ee:25:15:f2:39 Sandbox:} {Name:eth0 Mac:ae:11:aa:1d:20:4a Sandbox:/run/netns/netns-b3666979-98c8-d000-ed83-a27eb93dd5e4}] [{Version:4 Interface:0xc0002cd118 Address:{IP:10.88.0.10 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}] {[]  [] []}}"
time="2023-10-19T14:01:58Z" level=debug msg="Tearing down network namespace at /run/netns/netns-b3666979-98c8-d000-ed83-a27eb93dd5e4 for container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144"
time="2023-10-19T14:01:58Z" level=debug msg="Cleaning up container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144"
time="2023-10-19T14:01:58Z" level=debug msg="Network is already cleaned up, skipping..."
time="2023-10-19T14:01:58Z" level=debug msg="Container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144 storage is already unmounted, skipping..."
time="2023-10-19T14:01:58Z" level=debug msg="ExitCode msg: \"mounting storage for container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144: creating overlay mount to /var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/merged, mount_data=\\\"lowerdir=/var/lib/containers/storage/overlay/l/ozk4ftowyddqbvkgjfht7owizl,upperdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/diff,workdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/work,nodev\\\": using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first\\nfuse-overlayfs: cannot mount: no such file or directory\\n: exit status 1\""
Error: mounting storage for container d965ff95781a8d1be6d9e56d3c1f76cf907e88f245f8d4e4571a39344591c144: creating overlay mount to /var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/OZK4FTOWYDDQBVKGJFHT7OWIZL,upperdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/diff,workdir=/var/lib/containers/storage/overlay/2e00d5599cc052ff347b031dc8b2881f0b6312a12ccc63e736b80783642a8829/work,nodev": using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

DESCRIPTION
       fuse‐overlayfs provides an overlayfs FUSE implementation so that it can be used since Linux 4.18 by unprivileged users in an user namespace.

Given the description from fuse-overlayfs I bet this is not supported in 4.12.14-150100.197.157-default (15-sp1) as installed podman in the container uses overlayfs. On the host it uses btrfs driver

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-150500.9.6.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 96.42
    systemPercent: 0.84
    userPercent: 2.74
  cpus: 1
  distribution:
    distribution: '"sles"'
    version: "15.5"
  eventLogger: file
  hostname: 01a1e5c024c5
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.12.14-150100.197.157-default
  linkmode: dynamic
  logDriver: journald
  memFree: 62783488
  memTotal: 1019469824
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.8-150000.49.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.8
      commit: v1.1.8-0-g82f18fe0e44a
      spec: 1.0.2-dev
      go: go1.21.0
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-150500.1.1.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 2h 32m 56.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 0
    stopped: 9
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 25732034560
  graphRootUsed: 2933501952
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.4
  Built: 1680004800
  BuiltTime: Tue Mar 28 12:00:00 2023
  GitCommit: ""
  GoVersion: go1.18.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.4

host's podman-info

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.3-150100.3.12.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: unknown'
  cpus: 1
  distribution:
    distribution: '"sles"'
    version: "15.1"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.12.14-150100.197.157-default
  linkmode: dynamic
  memFree: 93220864
  memTotal: 1019469824
  ociRuntime:
    name: runc
    package: runc-1.1.9-150000.52.2.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.9
      commit: v1.1.9-0-gccaecfcbc907
      spec: 1.0.2-dev
      go: go1.21.3
      libseccomp: 2.4.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 2h 33m 23.47s (Approximately 0.08 days)
registries:
  search:
  - registry.opensuse.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Build Version: 'Btrfs v4.19.1 '
    Library Version: "102"
  imageStore:
    number: 1
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1603713600
  BuiltTime: Mon Oct 26 12:00:00 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18058

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18087

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/18103