Project

General

Profile

Actions

action #163790

open

OSD openqa.ini is corrupted, invalid characters

Added by okurz 16 days ago. Updated about 8 hours ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-07-10
Due date:
2024-08-10 (Due in 14 days)
% Done:

0%

Estimated time:

Description

Observation

I copied the corrupted config file to /etc/openqa/openqa.ini.corrupted-2024-07-11-okurz-poo163790

On backup-vm.qe.nue2.suse.org I see:

okurz@backup-vm:~> ls -la /home/rsnapshot/*/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul 11 19:32 /home/rsnapshot/alpha.0/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul 11 15:32 /home/rsnapshot/alpha.1/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul 11 12:32 /home/rsnapshot/alpha.2/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 2 martchus root 13056 Jul 11 07:32 /home/rsnapshot/alpha.3/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 2 martchus root 13056 Jul 11 07:32 /home/rsnapshot/alpha.4/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul 11 03:32 /home/rsnapshot/alpha.5/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul 10 03:32 /home/rsnapshot/beta.0/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  9 03:32 /home/rsnapshot/beta.1/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  8 03:32 /home/rsnapshot/beta.2/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  7 03:32 /home/rsnapshot/beta.3/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  6 03:32 /home/rsnapshot/beta.4/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  5 03:32 /home/rsnapshot/beta.5/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 13056 Jul  4 03:32 /home/rsnapshot/beta.6/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10259 Dec 17  2023 /home/rsnapshot/_delete.14764/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10267 Jan 21 11:32 /home/rsnapshot/_delete.15309/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root  1976 May 31 09:32 /home/rsnapshot/delta.0/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10312 Apr 26 03:33 /home/rsnapshot/delta.1/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10312 Mar 29 03:32 /home/rsnapshot/delta.2/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10463 Jun 28 03:11 /home/rsnapshot/gamma.0/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10463 Jun 21 03:11 /home/rsnapshot/gamma.1/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10463 Jun 14 03:32 /home/rsnapshot/gamma.2/openqa.suse.de/etc/openqa/openqa.ini
-rw-r--r-- 1 martchus root 10463 Jun  7 03:32 /home/rsnapshot/gamma.3/openqa.suse.de/etc/openqa/openqa.ini

so judging from the size it seems like 2024-06-28 is the last good. I copied back that config to OSD with

ssh backup-vm.qe.nue2.suse.org "cat /home/rsnapshot/gamma.0/openqa.suse.de/etc/openqa/openqa.ini" | ssh osd "cat - | sudo tee /etc/openqa/openqa.ini"

and restart the openqa-webui service.


Related issues 1 (0 open1 closed)

Copied from openQA Infrastructure - action #163592: [alert] (HTTP Response alert Salt tm0h5mf4k) size:MResolvedokurz2024-07-10

Actions
Actions #1

Updated by okurz 16 days ago

  • Copied from action #163592: [alert] (HTTP Response alert Salt tm0h5mf4k) size:M added
Actions #2

Updated by okurz 16 days ago

  • Description updated (diff)
Actions #3

Updated by okurz 16 days ago

  • Description updated (diff)
  • Due date deleted (2024-07-25)
  • Status changed from New to Blocked
  • Priority changed from Urgent to Normal

repair applied, blocking on #163592

Actions #4

Updated by okurz about 24 hours ago

  • Status changed from Blocked to In Progress

#163592 was resolved. I didn't see the same problem again. I assume if the system is under heavy load then if salt is called files become corrupted. I am researching if this is known elsewhere.

Actions #5

Updated by openqa_review about 8 hours ago

  • Due date set to 2024-08-10

Setting due date based on mean cycle time of SUSE QE Tools

Actions

Also available in: Atom PDF