Project

General

Profile

Actions
Copy link

action #162347

closed

Invalid journald configuration on OSD (any maybe other hosts) size:S

Added by nicksinger 12 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Low
Assignee:
gpathak
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
Due date:
2025-02-27
% Done:

0%

Estimated time:
Tags:
infra

Description

Observation

While checking osd-logs nicksinger encountered:

[    4.485011] systemd-journald[628]: /etc/systemd/journald.conf.d/journal_size.conf:3: Failed to parse size value, ignoring: 20%

So https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/2f714dc265be4e962409afc997c260e4dab42b1b/openqa/server.sls#L421 seems to be outdated and should be updated to limit journald again.

Acceptance criteria

  • AC1: ssh osd 'journalctl' does not show "Failed to parse size" errors anymore

Suggestions

  • Check what the correct syntax is, % may not be allowed or no longer accepted?
  • Apply the fix for OSD
  • Then check what machines are affected e.g. workers, and apply the according fix there as well, e.g. according state files
Actions
Copy link
 #1

Updated by okurz 12 months ago

  • Tags set to infra
  • Category set to Regressions/Crashes
  • Target version set to Ready
  • Parent task set to #162350
Actions
Copy link
 #2

Updated by okurz 11 months ago

  • Target version changed from Ready to Tools - Next
Actions
Copy link
 #3

Updated by okurz 10 months ago

  • Subject changed from Invalid journald configuration on OSD (any maybe other hosts) to Invalid journald configuration on OSD (any maybe other hosts) size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by okurz 4 months ago

  • Target version changed from Tools - Next to Ready
Actions
Copy link
 #5

Updated by gpathak 4 months ago

  • Assignee set to gpathak
Actions
Copy link
 #6

Updated by gpathak 4 months ago

The journald.conf man page mentions to Specify values in bytes or use K, M, G, T, P, E as units for the specified sizes (equal to 1024, 1024²,... bytes).

Changed the setting on OSD to keep 20% of 80GB storage free.

[Journal]
SystemMaxUse=80G
SystemKeepFree=16G
SystemMaxFileSize=1G
SystemMaxFiles=200
Actions
Copy link
 #8

Updated by gpathak 4 months ago

  • Status changed from Workable to In Progress
Actions
Copy link
 #9

Updated by gpathak 4 months ago

The specified journald configuration file is not present on any other worker except OSD

gpathak@openqa:~> sudo salt '*' cmd.run 'cat /etc/systemd/journald.conf.d/journal_size.conf'
s390zl12.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
s390zl13.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqa.suse.de:
    [Journal]
    SystemMaxUse=80G
    SystemKeepFree=16G
    SystemMaxFileSize=1G
    SystemMaxFiles=200
diesel.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
ada.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
baremetal-support-prg2.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqaworker18.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqaworker16.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqaw5-xen.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqaworker17.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
osiris-1.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
qesapworker-prg6.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
storage.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
netboot.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
qesapworker-prg7.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
qesapworker-prg4.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker40.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker34.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker36.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker29.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker31.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker35.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
sapworker1.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker33.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker-arm2.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker-arm1.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker39.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker30.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
unreal6.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
petrol.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
qesapworker-prg5.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
worker32.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqaworker14.qa.suse.cz:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
mania.qe.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
schort-server.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
openqa-piworker.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
baremetal-support.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
qamaster.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
backup-vm.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
jenkins.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
monitor.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
backup-qam.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
grenache-1.oqa.prg2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
tumblesle.qe.nue2.suse.org:
    cat: /etc/systemd/journald.conf.d/journal_size.conf: No such file or directory
Actions
Copy link
 #10

Updated by okurz 4 months ago

gpathak wrote in #note-6:

The journald.conf man page mentions to Specify values in bytes or use K, M, G, T, P, E as units for the specified sizes (equal to 1024, 1024²,... bytes).

Changed the setting on OSD to keep 20% of 80GB storage free.

[Journal]
SystemMaxUse=80G
SystemKeepFree=16G
SystemMaxFileSize=1G
SystemMaxFiles=200

https://manpath.be/c7/5/journald.conf#L129 mentions percent and I am sure those values worked and did not cause errors when we introduced them. Maybe the format changed and the character '%' shouldn't be used?

Actions
Copy link
 #11

Updated by gpathak 4 months ago · Edited

okurz wrote in #note-10:

gpathak wrote in #note-6:

The journald.conf man page mentions to Specify values in bytes or use K, M, G, T, P, E as units for the specified sizes (equal to 1024, 1024²,... bytes).

Changed the setting on OSD to keep 20% of 80GB storage free.

[Journal]
SystemMaxUse=80G
SystemKeepFree=16G
SystemMaxFileSize=1G
SystemMaxFiles=200

https://manpath.be/c7/5/journald.conf#L129 mentions percent and I am sure those values worked and did not cause errors when we introduced them. Maybe the format changed and the character '%' shouldn't be used?

I was also confused at first, but the line says

The first pair defaults to 10% and the second to 15% of the size of the respective file system , but each value is capped to 4G

which is the default value that journald will consider if not specified or not found explicitly in any config file

Actions
Copy link
 #12

Updated by okurz 4 months ago

So systemd can compute relative values but one can't enter them? I am pretty sure I tested that when I introduced it, strange.

Actions
Copy link
 #13

Updated by gpathak 4 months ago

okurz wrote in #note-12:

So systemd can compute relative values but one can't enter them? I am pretty sure I tested that when I introduced it, strange.

Please refer:

Actions
Copy link
 #14

Updated by gpathak 4 months ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #15

Updated by okurz 4 months ago

  • Due date set to 2025-02-27
Actions
Copy link
 #17

Updated by gpathak 3 months ago

  • Status changed from Feedback to Resolved

okurz wrote in #note-16:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1373 merged.

PR merged,resolving the ticket.

Actions
Copy link
 #18

Updated by okurz 3 months ago

  • Parent task deleted (#162350)
Actions
Copy link

Also available in: Atom PDF