Project

General

Profile

Actions

action #152827

closed

[tools] cron service updating clamav database failing on OSD + O3 size:S

Added by osukup 6 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2023-12-20
Due date:
% Done:

0%

Estimated time:

Description

Observation

From /var/spool/mail/cvdupdate on both instances of the osd and o3 web UI:

From cvdupdate@localhost  Thu Dec 21 10:00:01 2023
Return-Path: <cvdupdate@localhost>
X-Original-To: cvdupdate
Delivered-To: cvdupdate@localhost
Received: by localhost (Postfix, from userid 17307)
        id BC86134590; Thu, 21 Dec 2023 10:00:01 +0100 (CET)
From: "(Cron Daemon)" <cvdupdate@localhost>
To: cvdupdate@localhost
Subject: Cron <cvdupdate@openqa> /home/cvdupdate/.local/bin/cvdupdate update
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Auto-Submitted: auto-generated
Precedence: bulk
X-Cron-Env: <XDG_SESSION_ID=c21238>
X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/17307>
X-Cron-Env: <DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/17307/bus>
X-Cron-Env: <XDG_SESSION_TYPE=unspecified>
X-Cron-Env: <XDG_SESSION_CLASS=background>
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/cvdupdate>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=cvdupdate>
X-Cron-Env: <USER=cvdupdate>
Message-Id: <20231221090001.BC86134590@localhost>
Date: Thu, 21 Dec 2023 10:00:01 +0100 (CET)

Traceback (most recent call last):
  File "/home/cvdupdate/.local/bin/cvdupdate", line 11, in <module>
    sys.exit(cli())
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 259, in update_alias
    ctx.forward(db_update)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 100, in db_update
    m = CVDUpdate(config=config, verbose=verbose)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 119, in __init__
    nameserver)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 184, in _read_config
    self.config = json.load(config_file)
  File "/usr/lib64/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Acceptance criteria

  • AC1: Clamav is no longer raising exceptions on o3
  • AC2: Clamav is no longer raising exceptions on osd

Suggestions

  • Remove clamav service - install on o3, remove from salt on osd

Related issues 1 (0 open1 closed)

Related to openQA Tests - action #89401: [qe-core][clamav] create local mirror for CVD updates auto_review:"console/clamav.pm line 55."Resolveddzedro2021-03-03

Actions
Actions #1

Updated by osukup 6 months ago

  • Related to action #89401: [qe-core][clamav] create local mirror for CVD updates auto_review:"console/clamav.pm line 55." added
Actions #2

Updated by osukup 6 months ago

  • Subject changed from [tools] cron service updating clamav database failing on OSD to [tools] cron service updating clamav database failing on OSD + O3
  • Description updated (diff)
Actions #3

Updated by okurz 6 months ago

  • Tags set to infra, reactive work
  • Target version set to Ready
Actions #4

Updated by livdywan 6 months ago

  • Subject changed from [tools] cron service updating clamav database failing on OSD + O3 to [tools] cron service updating clamav database failing on OSD + O3 size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by dheidler 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions #6

Updated by dheidler 6 months ago · Edited

The error is raised from a tool that was set up in https://progress.opensuse.org/issues/89401 to maintain a local mirror of the database for the clamav tests run in openqa.

For whatever reason the config.json was empty on both servers, so it had to be recreated:

ssh openqa.suse.de
cat /etc/cron.d/cvd
sudo su cvdupdate
rm ~/.cvdupdate/config.json # remove empty config
~/.local/bin/cvdupdate config set --dbdir /var/lib/openqa/factory/repo/cvd/ # create new config
Actions #7

Updated by dheidler 6 months ago

This started to fail on 27.September 15:00 CEST:

Date: Wed, 27 Sep 2023 15:00:02 +0200 (CEST)
From: "(Cron Daemon)" <cvdupdate@localhost>
To: cvdupdate@localhost
Subject: Cron <cvdupdate@openqa> /home/cvdupdate/.local/bin/cvdupdate update

2023-09-27 15:00:02 cvdupdate-0.3.0 INFO main.cvd is up-to-date. Version: 62
2023-09-27 15:00:02 cvdupdate-0.3.0 INFO daily.cvd is up-to-date. Version: 27044
2023-09-27 15:00:02 cvdupdate-0.3.0 INFO bytecode.cvd is up-to-date. Version: 334
Failed to create config file!
Traceback (most recent call last):
  File "/home/cvdupdate/.local/bin/cvdupdate", line 11, in <module>
    sys.exit(cli())
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 259, in update_alias
    ctx.forward(db_update)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 628, in forward
    return self.invoke(cmd, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/__main__.py", line 101, in db_update
    errors = m.db_update(db, debug_mode)
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 876, in db_update
    self._save_config()
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 237, in _save_config
    raise exc
  File "/home/cvdupdate/.local/lib/python3.6/site-packages/cvdupdate/cvdupdate.py", line 234, in _save_config
    json.dump(self.config, config_file, indent=4)
  File "/usr/lib64/python3.6/json/__init__.py", line 180, in dump
    fp.write(chunk)
OSError: [Errno 28] No space left on device

For whatever reason this tool rewrites its config on every call. (Maybe to remember, what files are already downloaded).
And at that time the disk was full (on osd), so it couldn't be written.

Actions #8

Updated by dheidler 6 months ago · Edited

On o3 it was broken since 08.11.2021.

Actions #9

Updated by dheidler 6 months ago

I reduced the cron to daily instead of 15 minutes.
The script annoyingly writes to stderr even on success, so cron always writes an email.

Actions #10

Updated by dheidler 6 months ago · Edited

Added this to cron to delete old mails:

37 5 30 * * cvdupdate mutt -f /var/spool/mail/cvdupdate -e "push D~d>12m<enter>qy<enter>" > /dev/null
Actions #11

Updated by dheidler 6 months ago

  • Status changed from In Progress to Resolved

Should work again.

Actions

Also available in: Atom PDF