QA » openQA Project

Observation¶

From 2022-12-31:

[2022-12-31T16:00:05.500773Z] [error] [pid:1593] Unexpected error when updating job 3001504 executed by worker openqaworker4:1: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148

[2022-12-31T16:00:05.545694Z] [error] [pid:18320] Unexpected error when updating job 3001833 executed by worker openqaworker7:3: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.path, me.name FROM needle_dirs me WHERE ( me.path = ? )" with ParamValues: 1='/var/lib/openqa/share/tests/opensuse/products/opensuse/needles'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 114

[2022-12-31T16:00:05.549336Z] [error] [pid:31119] Unexpected error when updating job 3001427 executed by worker openqaworker4:16: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148

[2022-12-31T16:00:05.549414Z] [error] [pid:10945] Unexpected error when updating job 3001372 executed by worker openqaworker1:6: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.path, me.name FROM needle_dirs me WHERE ( me.path = ? )" with ParamValues: 1='/var/lib/openqa/share/tests/opensuse/products/opensuse/needles'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 114
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159

[2022-12-31T16:00:05.556698Z] [error] [pid:5585] Unexpected error when updating job 3001595 executed by worker openqaworker19:16: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='root-console-20180724.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159

[2022-12-31T16:00:05.557836Z] [error] [pid:21962] Unexpected error when updating job 3000999 executed by worker openqaworker4:20: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='inst-overview-kde-20180807.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159

[2022-12-31T16:00:05.562536Z] [error] [pid:18715] Unexpected error when updating job 3001815 executed by worker openqaworker1:7: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='gnome-terminal-LIVE-20210215.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159

[2022-12-31T16:00:05.565493Z] [error] [pid:2289] Unexpected error when updating job 3000826 executed by worker openqaworker7:4: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='inst-packageinstallationstarted-simplified_UI-20220113.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 155

[2022-12-31T16:00:05.567371Z] [error] [pid:31396] Unexpected error when updating job 3000490 executed by worker ip-10-252-32-98:1: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: FATAL:  terminating connection due to administrator command
server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "UPDATE jobs SET skipped_module_count = skipped_module_count - 1, t_updated = ? WHERE id = ?" with ParamValues: 1='2022-12-31 16:00:05', 2='3000490'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1236

[2022-12-31T16:00:05.567708Z] [error] [pid:29288] Unexpected error when updating job 3001223 executed by worker openqaworker4:15: Transaction aborted: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. [for Statement "SELECT me.id, me.dir_id, me.filename, me.last_seen_time, me.last_seen_module_id, me.last_matched_time, me.last_matched_module_id, me.last_updated, me.file_present, me.tags, me.t_created, me.t_updated FROM needles me WHERE ( ( me.dir_id = ? AND me.filename = ? ) )" with ParamValues: 1='9', 2='displaymanager-gdm-user-prompt-20201120.json'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 123
 Rollback failed: DBIx::Class::Storage::DBI::_exec_txn_rollback(): DBI Exception: DBD::Pg::db rollback failed: no connection to the server at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 159

[2022-12-31T16:00:05.600082Z] [error] [zaeBwZ9Z_q6u] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…
[2022-12-31T16:00:05.620697Z] [error] [pid:1941] Unexpected error when updating job 3000919 executed by worker openqaworker7:6: DBIx::Class::Storage::DBI::_exec_txn_commit(): DBI Exception: DBD::Pg::db commit failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Needles.pm line 148
…
[2022-12-31T16:00:05.647238Z] [error] [pid:28013] Unexpected error when updating job 3001883 executed by worker openqaworker4:8: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1460
…

[2022-12-31T16:00:05.669141Z] [error] [ZhAdj8_QHmOV] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:05.671175Z] [warn] [pid:11823] Unable to verify whether worker 514 runs its job(s) as expected: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebSockets/Controller/Worker.pm line 181

[2022-12-31T16:00:05.680413Z] [error] [RCvMdhcWEnLh] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:05.680868Z] [error] [pid:27335] Unexpected error when updating job 3001586 executed by worker openqaworker7:10: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1223



[2022-12-31T16:00:05.688380Z] [error] [pid:2613] Unexpected error when updating job 3001831 executed by worker qa-power8-3:1: DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobModules.pm line 183

[2022-12-31T16:00:05.689907Z] [error] [V-JN1psyxnSy] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…

[2022-12-31T16:00:05.764380Z] [error] [QAD4DnnQvmrW] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  the database system is shutting down at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/Jobs.pm line 1253
…
[2022-12-31T16:00:05.875734Z] [error] [qsQUUg43-z3n] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.result_dir, me.archived, me.state, me.priority, me.result, me.reason, me.clone_id, me.blocked_by_id, me.backend_info, me.TEST, me.DISTRI, me.VERSION, me.FLAVOR, me.ARCH, me.BUILD, me.MACHINE, me.group_id, me.assigned_worker_id, me.t_started, me.t_finished, me.logs_present, me.passed_module_count, me.failed_module_count, me.softfailed_module_count, me.skipped_module_count, me.externally_skipped_module_count, me.scheduled_product_id, me.result_size, me.t_created, me.t_updated, settings.id, settings.key, settings.value, settings.job_id, settings.t_created, settings.t_updated FROM jobs me LEFT JOIN job_settings settings ON settings.job_id = me.id WHERE ( me.id = ? )  ORDER BY me.id" with ParamValues: 1='3001829'] at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/API/V1/Job.pm line 43

[2022-12-31T16:00:06.014599Z] [error] [EGY0ZCyWByyc] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.015322Z] [error] [_o0mlLhZa-Fj] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.019578Z] [error] [iotRxnhwRJmT] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.027060Z] [error] [YmUHBsS_dwlQ] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request. at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Test.pm line 569

[2022-12-31T16:00:06.080922Z] [error] [YrL-CwJSnIfB] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.112139Z] [error] [Zujueq11y7rK] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
    Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.226838Z] [error] [Bkw6wx2Dz7i_] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
    Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

[2022-12-31T16:00:06.231040Z] [error] [3s2hW8oTou9V] DBIx::Class::Storage::DBI::catch {...} (): DBI Connection failed: DBI connect('dbname=openqa','geekotest',...) failed: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
    Is the server running locally and accepting connections on that socket? at /usr/lib/perl5/vendor_perl/5.26.1/DBIx/Class/Storage/DBI.pm line 1517. at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/API/V1/Job.pm line 433

[2022-12-31T16:00:06.248453Z] [error] [XtcnY7RUnh67] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145
…
[2022-12-31T16:00:11.772881Z] [error] [z8hqls9h70p-] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: no connection to the server [for Statement "SELECT me.id, me.key, me.secret, me.user_id, me.t_expiration, me.t_created, me.t_updated FROM api_keys me WHERE ( me.key = ? )" with ParamValues: 1='XXX'] at /usr/share/openqa/script/../lib/OpenQA/Shared/Controller/Auth.pm line 145

Acceptance criteria¶

• AC1: o3 works without error messages in logs when the database server is shutdown on purpose
• AC2: There are still error messages if a database server is not accessible for longer

Suggestions¶

• Check log if problem persists including postgres logs
• Check systemd service files in the case of database service shutdown to ensure that services relying on the database are stopped first, e.g. ensure that if one calls systemctl stop postgres then also openQA stops or something