[MDEV-29293] MariaDB stuck on starting commit state (waiting on commit order critical section) - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.5.15, 10.6.12
Fix Version/s: 10.4.31, 10.5.22, 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3, 11.1.2
Component/s: Galera
Labels:
None
Environment:
Galera version: 26.4.6-buster
Debian Version: 10.6
Kernel version: 4.19.0-12-amd64

Description

In an environment running Galera Cluster with 6 MariaDB nodes, 1 arbitrator node, some replicas and a ProxySQL, after a network issue that triggered a state transfer on two nodes,
for some reason, almost all the transactions hang in:

“starting” state on the commit statement or on "".
"acquiring total order isolation" on the "KILL CONNECTION" statement (The "KILL CONNECTION" was requested by the ProxySQL)
We tried to restart the service but it hangs on stopping, ProxySQL detected this node as down and switched the traffic to another node.

By looking at the backtrace it seems that we have a kind of "pthread_cond_wait() deadlock" executed by lock.wait() on the enter() function on the commit monitor during the commit order critical section.

Unfortunately, we didn't find a way to reproduce the problem

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

backtraces.txt
315 kB
2022-08-11 11:24
innodb_status.txt
67 kB
2022-08-11 11:24
process_list.txt
467 kB
2022-08-11 11:24
processlist.png
701 kB
2022-10-14 01:18
process-list-sample.txt
2 kB
2022-09-01 16:00

Issue Links

blocks

MDEV-30963 Assertion failure !lock.was_chosen_as_deadlock_victim in trx0trx.h:1065

Closed

causes

MDEV-29346 update_rows_log_event hung causing galera cluster failure

Closed

MDEV-30372 Assertion `state() == s_executing || state() == s_preparing || state() == s_prepared || state() == s_must_abort || state() == s_aborting || state() == s_cert_failed || state() == s_must_replay' failed

Closed

includes

MDEV-31075 KILL QUERY maintains nodes data consistency but breaks GTID sequence

Closed

relates to

MDEV-28472 BF lock wait long for trx - Assertion `mode_ == m_local || transaction_.is_streaming()' failed

Closed

MDEV-29323 Galera ha_abort_transaction is not honored if there are no InnoDB lock conflicts

Open

(1 relates to)

Activity

Ascending order - Click to sort in descending order

William Welter created issue - 2022-08-11 11:28

Marko Mäkelä made changes - 2022-08-11 22:45

Field	Original Value	New Value
Assignee		Jan Lindström [ jplindst ]

Carl Dobson made changes - 2022-09-01 16:00

Attachment

process-list-sample.txt [ 65246 ]

Jan Lindström (Inactive) made changes - 2022-09-30 08:15

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Khai Ping made changes - 2022-10-14 01:18

Attachment

processlist.png [ 65721 ]

Elena Stepanova made changes - 2022-11-23 15:58

Fix Version/s

10.5 [ 23123 ]

Alasdair Haswell made changes - 2023-02-03 09:51

Attachment		gdb.txt_test3_100insertQPS.gz [ 68071 ]
Attachment		gdb.txt_test1.gz [ 68072 ]
Attachment		gdb.txt_test2_200insertQPS.gz [ 68073 ]

Seppo Jaakola made changes - 2023-02-06 09:28

Status

Open [ 1 ]

In Progress [ 3 ]

Seppo Jaakola made changes - 2023-02-06 09:49

Status

In Progress [ 3 ]

Needs Feedback [ 10501 ]

Alasdair Haswell made changes - 2023-02-07 14:10

Attachment		gdb_010.txt.gz [ 68120 ]
Attachment		gdb_008.txt.gz [ 68121 ]
Attachment		gdb_007.txt.gz [ 68122 ]
Attachment		gdb_006.txt.gz [ 68123 ]

Alasdair Haswell made changes - 2023-02-08 14:30

Attachment		mariadb_003.err.gz [ 68164 ]
Attachment		mariadb_001.err.gz [ 68165 ]
Attachment		gdb.txt_003.gz [ 68166 ]
Attachment		gdb.txt_002.gz [ 68167 ]

Chris Calender (Inactive) made changes - 2023-02-08 16:18

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Julien Fritsch made changes - 2023-02-08 17:19

Status

Open [ 1 ]

Needs Feedback [ 10501 ]

Alasdair Haswell made changes - 2023-02-09 10:14

Attachment

oltp_insert_nba.lua.rtf [ 68189 ]

John Gehring made changes - 2023-02-10 01:36

Attachment

gdb_007.txt.gz [ 68122 ]

John Gehring made changes - 2023-02-10 01:36

Attachment

gdb_008.txt.gz [ 68121 ]