action #161309
Updated by okurz about 2 months ago
## Observation Users pointed out in https://suse.slack.com/archives/C02CANHLANP/p1717141066882819 that osd is down, the problem was confirmed. As the nginx was complaining about its upstream, openqa-webui.service was restarted to remedy the situation. After that, the following error was seen in the logs ``` May 31 09:39:52 openqa openqa[19648]: Error when trying to get the database version: DBIx::Class::Storage::DBI::_do_query(): DBI Exception: DBD::Pg::db do failed: ERROR: syntax error at or near "PRAGMA" May 31 09:39:52 openqa openqa[19648]: LINE 1: PRAGMA synchronous = OFF May 31 09:39:52 openqa openqa[19648]: ^ at inline delegation in DBIx::Class::DeploymentHandler::VersionStorage::Standard for version_rs->database_version (attribute declared in /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/DeploymentHandler/VersionStorage/Standard.pm at line> ``` No service restart helped. As there was also a glibc update involved helped, so we decided to ensure consistency with a do the full system reboot as commented in #161309-1 https://progress.opensuse.org/issues/161309#note-1 There was a deployment running https://gitlab.suse.de/openqa/salt-states-openqa/-/pipelines/1149031 just before the incident, the issue can be already seen in the telegraf post-deploy job. ``` openqa.suse.de: 2024-05-31T07:33:42Z E! [inputs.http] Error in plugin: [url=https://openqa.suse.de/admin/influxdb/jobs]: received status code 502 (Bad Gateway), expected any value out of [200] 2024-05-31T07:33:42Z E! [inputs.http] Error in plugin: [url=https://openqa.suse.de/admin/influxdb/minion]: received status code 502 (Bad Gateway), expected any value out of [200] 2024-05-31T07:33:45Z E! [telegraf] Error running agent: input plugins recorded 2 errors ```