QA (public) » openQA Project (public) » openQA Infrastructure (public)

The above data and even more insights are available as charts: https://zabbix.suse.de/zabbix.php?action=charts.view&filter_hostids%5B0%5D=10923&filter_show=1&filter_set=1

It seems we're keeping a long list of old packages in /var/cache/zypp/packages/. It goes back to february 2023:

ls -lrth /var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-*                                                                                                                             
-rw-r--r-- 1 root root 459K Feb 15  2023 /var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-4.6.1676474487.945e502-lp154.5577.1.x86_64.rpm                                                                

Not sure how to configure this to a lower duration.

We're keeping the packages of the following repos indefinitely:

grep -iR keeppackages=1 /etc/zypp/repos.d
/etc/zypp/repos.d/devel_openQA.repo:keeppackages=1
/etc/zypp/repos.d/devel_openQA_Leap.repo:keeppackages=1

Not sure whether zypper has a way of specifying the number of packages to keep. For now I just used `find /var/cache/zypp/packages -ipath 'devel_openqa' -mtime +365 -delete´ to delete everything older than a year.

I can setup a systemd service/timer to invoke a command like that periodically. I can also set keeppackages=0 but we probably enabled this for the sake of easier downgrades. So this is probably not a good solution.

One could also add the following to openqa-auto-update:

if [[ $OPENQA_PACKAGE_CACHE_RETENTION ]]; then
    find /var/cache/zypp/packages -type f -ipath '*devel*openQA*' -mtime "+$OPENQA_PACKAGE_CACHE_RETENTION" -delete
fi

Of course this breaks if one uses a different repository name or a different packagesdir. So it is probably not the best idea to add it to the generic openqa-auto-update script.

Let's block on #174316 before trying to adjust the numbers

A side note, /var/tmp looks to be actually the same filesystem as /, it seems that zabbix wrongly detected it twice.

and for the reference, it is a bug in the provided systemd unit, see https://github.com/voxpupuli/puppet-zabbix/issues/320 for more context. I adjusted the unit file to fix that issue.

jbaier_cz wrote in #note-16:

and for the reference, it is a bug in the provided systemd unit, see https://github.com/voxpupuli/puppet-zabbix/issues/320 for more context. I adjusted the unit file to fix that issue.

Could you write down here the change you made? I don't really get it.

@okurz

Well, as openqa-auto-update is openQA-specific, at least in the name, but also because it calls https://github.com/os-autoinst/openQA/blob/master/script/openqa-check-devel-repo which uses devel:openQA …

I looked into what we do so far as well. The problem is not that the new code is specific to openQA and our concrete packaging. The existing code is as well - which makes sense because it is part of the concrete packaging.

The problem is what I mentioned in #174313#note-9:

Of course this breaks if one uses a different repository name or a different packagesdir.

In other words, the new code is specific to the concrete local repository setup and zypper configuration. I considered making it read the relevant bits from the zypper config file but found it too involved.

I'm not sure what your code with tail … would achieve. However, I suppose it would indeed make sense to keep a certain number of copies instead of going by time. Since I've been using openSUSE I find the lack of a tool like paccache which I'm used to from Arch Linux (and MSYS2) and does exactly what you suggested quite annoying.

@okurz What about something like this?

openqa-clean-devel-repo-cache:

#!/bin/bash
set -e

OPENQA_PACKAGE_CACHE_RETENTION=${OPENQA_PACKAGE_CACHE_RETENTION:-100}
OPENQA_PACKAGE_CACHE_RETENTION_KEEP_MIN=${OPENQA_PACKAGE_CACHE_RETENTION_KEEP_MIN:-3}
OPENQA_PACKAGE_CACHE_PATH=${OPENQA_PACKAGE_CACHE_PATH:-/hdd/cache/zypp/packages}
OPENQA_PACKAGE_CACHE_REPO_GLOB=${OPENQA_PACKAGE_CACHE_REPO_GLOB:-'*devel*openQA*'}

IFS=$'\n'
package_files=($(find "$OPENQA_PACKAGE_CACHE_PATH" -type f -ipath "$OPENQA_PACKAGE_CACHE_REPO_GLOB" -mtime "+$OPENQA_PACKAGE_CACHE_RETENTION" | sort -rV))

previous_package_name=
package_count=0

for package_file in "${package_files[@]}"; do
    package_name=$(rpm -q --qf "%{NAME}\n" "$package_file")
    if [[ $package_name != "$previous_package_name" ]]; then
        previous_package_name=$package_name
        package_count=0
    fi
    package_count=$((package_count + 1))
    if [[ $package_count -gt "$OPENQA_PACKAGE_CACHE_RETENTION_KEEP_MIN" ]]; then
        echo "rm  " "$package_file"
    else
        echo "keep" "$package_file"
    fi
done

One could also remove the mtime parameter completely. The use of sort -rV should make sure that the newest packages survive. The use of rpm -q --qf "%{NAME}\n" "$package_file" helps to decide which package files are actually the same package (but just different versions).

This script produces sane output on my local system (also when removing the mtime parameter).

We still have to decide where to this script. I suppose we could add it to openqa-auto-update with all the specifics put into variables. I would make it so it doesn't run by default. We could however specify common defaults for OPENQA_PACKAGE_CACHE_PATH and OPENQA_PACKAGE_CACHE_RETENTION_KEEP_MIN.

mkittler wrote in #note-20:

@okurz What about something like this?

LGTM

One could also remove the mtime parameter completely.

I would keep the mtime as a safety measure.

We still have to decide where to this script. I suppose we could add it to openqa-auto-update with all the specifics put into variables. I would make it so it doesn't run by default. We could however specify common defaults for OPENQA_PACKAGE_CACHE_PATH and OPENQA_PACKAGE_CACHE_RETENTION_KEEP_MIN.

yes, all that sounds good.

tinita wrote in #note-18:

jbaier_cz wrote in #note-16:

and for the reference, it is a bug in the provided systemd unit, see https://github.com/voxpupuli/puppet-zabbix/issues/320 for more context. I adjusted the unit file to fix that issue.

Could you write down here the change you made? I don't really get it.

Sure, see systemctl cat zabbix_agentd.service, I just added a following snippet as recommended in the linked issue:

# /etc/systemd/system/zabbix_agentd.service.d/override.conf
[Service]
PrivateTmp=no

I also just had a look at one of the repo config files on openqaworker23 (/etc/zypp/repos.d/devel_openQA.repo) and I don't see that keeppackages=1 is configured. So a clean cache directory is supposedly expected on that machines (and probably others).