Project

General

Profile

Actions

action #121777

closed

PostgreSQL update ordering deadlock on jobs table

Added by kraih almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2022-12-09
Due date:
% Done:

0%

Estimated time:

Description

Observation

Seen in the O3 PostgreSQL logs:

2022-12-09 06:21:08.416 UTC openqa geekotest [6931]ERROR:  deadlock detected
2022-12-09 06:21:08.416 UTC openqa geekotest [6931]DETAIL:  Process 6931 waits for ShareLock on transaction 243609936; blocked by process 5779.
        Process 5779 waits for ShareLock on transaction 243609932; blocked by process 6931.
        Process 6931: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
        Process 5779: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
2022-12-09 06:54:32.659 UTC openqa geekotest [28685]ERROR:  deadlock detected
2022-12-09 06:54:32.659 UTC openqa geekotest [28685]DETAIL:  Process 28685 waits for ShareLock on transaction 243819997; blocked by process 30309.
        Process 30309 waits for ShareLock on transaction 243819999; blocked by process 28685.
        Process 28685: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
        Process 30309: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
2022-12-09 10:01:48.090 UTC openqa geekotest [21755]ERROR:  deadlock detected
2022-12-09 10:01:48.090 UTC openqa geekotest [21755]DETAIL:  Process 21755 waits for ShareLock on transaction 244660614; blocked by process 22220.
        Process 22220 waits for ShareLock on transaction 244660621; blocked by process 21755.
        Process 21755: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
        Process 22220: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3

This is probably a very similar problem to #120891, and i noticed it while checking the logs to verify the fix for that works.

Acceptance criteria

  • AC1: No more deadlocks on jobs table

Suggestions


Related issues 2 (0 open2 closed)

Related to openQA Project (public) - action #120891: Product not scheduled: DBD::Pg::st execute failed: ERROR: deadlock detected size:MResolvedmkittler2022-11-232023-01-20

Actions
Related to openQA Project (public) - action #121768: openQA jobs have been restarted but are stuck in "running" size:MResolvedmkittler2022-12-092023-01-19

Actions
Actions

Also available in: Atom PDF