action #122578
Updated by okurz almost 2 years ago
## Observation
From 2022-12-31:
```
[2022-12-31T16:00:05.500773Z] [error] [pid:1593] Unexpected error when updating job 3001504 executed by worker openqaworker4:1: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148
[2022-12-31T16:00:05.545694Z] [error] [pid:18320] Unexpected error when updating job 3001833 executed by worker openqaworker7:3: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.path, me.name FROM needle_dirs me WHERE ( me.path = ? )" with ParamValues: 1='/var/lib/openqa/share/tests/opensuse/products/opensuse/needles'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 114
[2022-12-31T16:00:05.549336Z] [error] [pid:31119] Unexpected error when updating job 3001427 executed by worker openqaworker4:16: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148
[2022-12-31T16:00:05.549414Z] [error] [pid:10945] Unexpected error when updating job 3001372 executed by worker openqaworker1:6: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.path, me.name FROM needle_dirs me WHERE ( me.path = ? )" with ParamValues: 1='/var/lib/openqa/share/tests/opensuse/products/opensuse/needles'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 114
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159
[2022-12-31T16:00:05.556698Z] [error] [pid:5585] Unexpected error when updating job 3001595 executed by worker openqaworker19:16: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='root-console-20180724.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159
[2022-12-31T16:00:05.557836Z] [error] [pid:21962] Unexpected error when updating job 3000999 executed by worker openqaworker4:20: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='inst-overview-kde-20180807.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159
[2022-12-31T16:00:05.562536Z] [error] [pid:18715] Unexpected error when updating job 3001815 executed by worker openqaworker1:7: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='gnome-terminal-LIVE-20210215.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159
[2022-12-31T16:00:05.565493Z] [error] [pid:2289] Unexpected error when updating job 3000826 executed by worker openqaworker7:4: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='inst-packageinstallationstarted-simplified_UI-20220113.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 155
[2022-12-31T16:00:05.567371Z] [error] [pid:31396] Unexpected error when updating job 3000490 executed by worker ip-10-252-32-98:1: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL: terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "UPDATE jobs SET skipped_module_count = skipped_module_count - 1, t_updated = ? WHERE id = ?" with ParamValues: 1='2022-12-31 16:00:05', 2='3000490'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1236
[2022-12-31T16:00:05.567708Z] [error] [pid:29288] Unexpected error when updating job 3001223 executed by worker openqaworker4:15: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='displaymanager-gdm-user-prompt-20201120.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159
[2022-12-31T16:00:05.600082Z] [error] [zaeBwZ9Z_q6u] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…
[2022-12-31T16:00:05.620697Z] [error] [pid:1941] Unexpected error when updating job 3000919 executed by worker openqaworker7:6: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148
…
[2022-12-31T16:00:05.647238Z] [error] [pid:28013] Unexpected error when updating job 3001883 executed by worker openqaworker4:8: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1460
…
[2022-12-31T16:00:05.669141Z] [error] [ZhAdj8_QHmOV] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:05.671175Z] [warn] [pid:11823] Unable to verify whether worker 514 runs its job(s) as expected: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebSockets/Controller/Worker.pm line 181
[2022-12-31T16:00:05.680413Z] [error] [RCvMdhcWEnLh] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:05.680868Z] [error] [pid:27335] Unexpected error when updating job 3001586 executed by worker openqaworker7:10: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1223
[2022-12-31T16:00:05.688380Z] [error] [pid:2613] Unexpected error when updating job 3001831 executed by worker qa-power8-3:1: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 183
[2022-12-31T16:00:05.689907Z] [error] [V-JN1psyxnSy] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…
[2022-12-31T16:00:05.764380Z] [error] [QAD4DnnQvmrW] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1253
…
[2022-12-31T16:00:05.875734Z] [error] [qsQUUg43-z3n] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.result_dir, me.archived, me.state, me.priority, me.result, me.reason, me.clone_id, me.blocked_by_id, me.backend_info, me.TEST, me.DISTRI, me.VERSION, me.FLAVOR, me.ARCH, me.BUILD, me.MACHINE, me.group_id, me.assigned_worker_id, me.t_started, me.t_finished, me.logs_present, me.passed_module_count, me.failed_module_count, me.softfailed_module_count, me.skipped_module_count, me.externally_skipped_module_count, me.scheduled_product_id, me.result_size, me.t_created, me.t_updated, settings.id, settings.key, settings.value, settings.job_id, settings.t_created, settings.t_updated FROM jobs me LEFT JOIN job_settings settings ON settings.job_id = me.id WHERE ( me.id = ? ) ORDER BY me.id" with ParamValues: 1='3001829'] at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/API/V1/Job.pm line 43
[2022-12-31T16:00:06.014599Z] [error] [EGY0ZCyWByyc] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.015322Z] [error] [_o0mlLhZa-Fj] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.019578Z] [error] [iotRxnhwRJmT] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.027060Z] [error] [YmUHBsS_dwlQ] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Test.pm line 569
[2022-12-31T16:00:06.080922Z] [error] [YrL-CwJSnIfB] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.112139Z] [error] [Zujueq11y7rK] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.226838Z] [error] [Bkw6wx2Dz7i_] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
[2022-12-31T16:00:06.231040Z] [error] [3s2hW8oTou9V] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/API/V1/Job.pm line 433
[2022-12-31T16:00:06.248453Z] [error] [XtcnY7RUnh67] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…
[2022-12-31T16:00:11.772881Z] [error] [z8hqls9h70p-] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
```
## Acceptance criteria
* **AC1:** o3 works without error messages in logs when the database about problems to connect to internal postgres server is shutdown on purpose
* **AC2:** There are still our code can handle without error messages if a the database server is temporarily not accessible for longer
## Suggestions
* Check log if problem persists including postgres logs
* Check systemd service files in the case of database service shutdown to ensure that services relying on the database are stopped first, e.g. ensure that if one calls `systemctl stop postgres` then also openQA stops or something first