Project

General

Profile

Actions
Copy link

action #177048

closed

Aggregates only on 12-SP5 x86_64 randomly don't get scheduled because of deadlock size:M

Added by dzedro about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2025-02-12
Due date:
2025-02-28
% Done:

0%

Estimated time:
Tags:
reactive work

Description

Lately sometimes 12-SP5 x86_64 are not scheduled, when I restart the schedule there is no problem.
At the time of scheduling there is some deadlock, very strange this happens only on 12-SP5 x86_64
Is there some conflict between Server-DVD-Updates-LTSS-ES & Server-DVD-Updates ?

https://openqa.suse.de/admin/productlog?id=2689877 BUILD=20250211-1
https://openqa.suse.de/admin/productlog?id=2691469 BUILD=20250212-1

{
    "failed_job_info": [
        {
            "error_message": "OpenQA::Schema::Result::Jobs::register_assets_from_settings(): DBI Exception: DBD::Pg::st execute failed: ERROR:  deadlock detected\nDETAIL:  Process 8553 waits for ShareLock on transaction 2490096299; blocked by process 21248.\nProcess 21248 waits for ShareLock on transaction 2490096379; blocked by process 8553.\nHINT:  See server log for query details.\nCONTEXT:  while inserting index tuple (795,49) in relation \"assets\" [for Statement \"INSERT INTO assets (type, name, t_created, t_updated)\n            VALUES (?,    ?,    now(),     now())\n    ON CONFLICT DO NOTHING RETURNING id\n\" with ParamValues: 1='hdd', 2='SLES-12-SP5-x86_64-mru-install-desktop-with-addons-Build20250211-1.qcow2'] at /usr/share/openqa/script/../lib/OpenQA/Schema/ResultSet/Jobs.pm line 205\n",
            "job_name": "qam-smt-server"
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732301
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732307
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732311
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732319
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732326
        },
        {
            "error_messages": [
                "DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
            ],
            "job_id": 16732331
        }
    ],
    "notes": [
        "Transaction failed: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  insert or update on table \"job_dependencies\" violates foreign key constraint \"job_dependencies_fk_child_job_id\"\nDETAIL:  Key (child_job_id)=(16732338) is not present in table \"jobs\". [for Statement \"INSERT INTO job_dependencies ( child_job_id, dependency, parent_job_id) VALUES ( ?, ?, ? )\" with ParamValues: 1='16732338', 2='1', 3='16732331'] at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/ScheduledProducts.pm line 703\n"
    ],
    "successful_job_ids": []
}

Acceptance criteria

  • AC1: Multiple products can be scheduled reliably at the same time

Suggestions

  • It looks like jobs are not created because scheduling fails within a transaction
  • This might happen when many products are scheduled at the same time?
  • Confirm how often this happens
  • Check for similar issues we had in the past and how the solution looked like back then. Maybe something was overlooked or now a similar problem surfaces at a slightly different place.
Actions
Copy link

Also available in: Atom PDF