action #112916
closedopenQA Project (public) - coordination #109846: [epic] Ensure all our database tables accomodate enough data, e.g. bigint for ids
postgresql.conf is invalid after recent salt changes size:M
0%
Description
Observation¶
- The current /srv/PSQL/data/postgresql.conf is invalid and contains lines and parts of lines containing "tMB".
- Postgres was last restarted yesterday i.e.
active (running) since Wed 2022-06-22 15:54:59 CEST; 19h agoacording to systemd meaning it was not restarted after the config changes - This seems to have been introduced by https://gitlab.suse.de/openqa/salt-states-openqa/-/commit/606eb48d1c54e25d6528c45dd230cf02d815f3d4
- All pipelines succeeded
- The config was last changed this morning, and there is an identical .bak file, and we don't know why:
-rw------- 1 postgres postgres 29374 Jun 23 10:47 postgresql.conf -rw------- 1 root root 29374 Jun 23 09:48 postgresql.conf.bak
Suggestions¶
- Observe the problem in https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1026706#L1433
- Fix the regex
- Verify validity of the config, e.g. a grep of the config file in gitlab CI or following https://stackoverflow.com/a/55758799 within the running server, that is running
select pg_reload_conf();
Updated by livdywan over 2 years ago
- Copied from coordination #112718: [alert][osd] openqa.suse.de is not reachable anymore, response times > 30s, multiple alerts over the weekend added
Updated by tinita over 2 years ago
- Subject changed from posttgresql.conf is invalid after recent salt changes size:M to postgresql.conf is invalid after recent salt changes size:M
Updated by okurz over 2 years ago
- Tracker changed from coordination to action
- Status changed from New to In Progress
manually reverting the config file from https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/1026706#L1434 and restarted the postgresql service. The file was shortly overwritten again, likely by salt, but nevermind because I will fix that soon in the salt repo. But in the broken state I called in psql select pg_reload_conf(); which still shows "t" despite the config being invalid. This is in line with https://stackoverflow.com/questions/16333319/how-to-syntax-check-postgresql-config-files#comment101836284_55758799 which also states that this does not work. Also https://stackoverflow.com/a/16340569 states that there is no separate check config command. One needs to try to restart the server which refuses to load an invalid config.
https://github.com/os-autoinst/salt-states-openqa/blob/master/openqa/server.sls#L330 is within a block {%- if not grains.get('noservices', False) %} so we won't see those rules effective in CI jobs.
Updated by okurz over 2 years ago
- Status changed from In Progress to Feedback
Manually testing the changes on OSD in a separate file with
salt-call --local -l error --state-output=changes state.sls_id postgresql-work_mem2 openqa.postgresql test=True
which takes about 10s each run – why is salt so slow – but ok.
Trying out the pattern manually with grep or sed is fine but with salt the replacement ends up as tMB
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/705
Updated by okurz over 2 years ago
- Due date deleted (
2022-07-07) - Status changed from Feedback to Resolved
We crosschecked and it's all good