action #177048
Updated by livdywan about 2 months ago
Lately sometimes 12-SP5 x86_64 are not scheduled, when I restart the schedule there is no problem.
At the time of scheduling there is some deadlock, very strange this happens only on 12-SP5 x86_64
Is there some conflict between Server-DVD-Updates-LTSS-ES & Server-DVD-Updates ?
https://openqa.suse.de/admin/productlog?id=2689877 BUILD=20250211-1
https://openqa.suse.de/admin/productlog?id=2691469 BUILD=20250212-1
```
{
"failed_job_info": [
{
"error_message": "OpenQA::Schema::Result::Jobs::register_assets_from_settings(): DBI Exception: DBD::Pg::st execute failed: ERROR: deadlock detected\nDETAIL: Process 8553 waits for ShareLock on transaction 2490096299; blocked by process 21248.\nProcess 21248 waits for ShareLock on transaction 2490096379; blocked by process 8553.\nHINT: See server log for query details.\nCONTEXT: while inserting index tuple (795,49) in relation \"assets\" [for Statement \"INSERT INTO assets (type, name, t_created, t_updated)\n VALUES (?, ?, now(), now())\n ON CONFLICT DO NOTHING RETURNING id\n\" with ParamValues: 1='hdd', 2='SLES-12-SP5-x86_64-mru-install-desktop-with-addons-Build20250211-1.qcow2'] at /usr/share/openqa/script/../lib/OpenQA/Schema/ResultSet/Jobs.pm line 205\n",
"job_name": "qam-smt-server"
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732301
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732307
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732311
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732319
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732326
},
{
"error_messages": [
"DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"job_id": 16732331
}
],
"notes": [
"Transaction failed: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL: Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
],
"successful_job_ids": []
}
```
## Acceptance criteria
* **AC1**: Multiple products can be scheduled reliably at the same time
## Suggestions
* It looks like jobs are not created because scheduling fails within a transaction
* This might happen when many products are scheduled at the same time?
* Confirm how often this happens
* Check for similar issues we had in the past and how the solution looked like back then. Maybe something was overlooked or now a similar problem surfaces at a slightly different place.