Actions
action #121777
closedPostgreSQL update ordering deadlock on jobs table
Description
Observation¶
Seen in the O3 PostgreSQL logs:
2022-12-09 06:21:08.416 UTC openqa geekotest [6931]ERROR: deadlock detected
2022-12-09 06:21:08.416 UTC openqa geekotest [6931]DETAIL: Process 6931 waits for ShareLock on transaction 243609936; blocked by process 5779.
Process 5779 waits for ShareLock on transaction 243609932; blocked by process 6931.
Process 6931: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
Process 5779: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
2022-12-09 06:54:32.659 UTC openqa geekotest [28685]ERROR: deadlock detected
2022-12-09 06:54:32.659 UTC openqa geekotest [28685]DETAIL: Process 28685 waits for ShareLock on transaction 243819997; blocked by process 30309.
Process 30309 waits for ShareLock on transaction 243819999; blocked by process 28685.
Process 28685: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
Process 30309: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
2022-12-09 10:01:48.090 UTC openqa geekotest [21755]ERROR: deadlock detected
2022-12-09 10:01:48.090 UTC openqa geekotest [21755]DETAIL: Process 21755 waits for ShareLock on transaction 244660614; blocked by process 22220.
Process 22220 waits for ShareLock on transaction 244660621; blocked by process 21755.
Process 21755: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
Process 22220: UPDATE jobs SET result = $1, t_updated = $2 WHERE id = $3
This is probably a very similar problem to #120891, and i noticed it while checking the logs to verify the fix for that works.
Acceptance criteria¶
- AC1: No more deadlocks on jobs table
Suggestions¶
- Try to get the jobs table updates into a consistent order across all processes (similar to https://progress.opensuse.org/issues/120891#note-21)
Actions