[MDEV-31658] Deadlock found when trying to get lock during applying - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.6.14
Fix Version/s: 10.6.19, 10.11.9, 11.1.6, 11.2.5, 11.4.3
Component/s: Galera
Labels:
None

Description

cluster got frozen after joiner node hit the following error:

2023-07-10  2:20:50 10 [ERROR] Slave SQL: Could not execute Write_rows_compressed_v1 event on table db.tablex; Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos 1583, Internal MariaDB error code: 1213

2023-07-10  2:20:50 103 [ERROR] Slave SQL: Could not execute Write_rows_v1 event on table db.tablex; Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos 772, Internal MariaDB error code: 1213

2023-07-10  2:20:50 10 [Warning] WSREP: Event 3 Write_rows_compressed_v1 apply failed: 149, seqno 45474182196

2023-07-10  2:20:50 103 [Warning] WSREP: Event 3 Write_rows_v1 apply failed: 149, seqno 45474182201

2023-07-10  2:20:50 0 [Note] WSREP: Member 0(fx114_db3) initiates vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,8b5f5fd24a23838b:  Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213;

2023-07-10  2:20:50 0 [Note] WSREP: Member 2(fx114_db2) responds to vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,0000000000000000: Success

2023-07-10  2:20:50 0 [Warning] WSREP: Received bogus VOTE message: 45474182196.0, from node f3b6f9e7-1ea8-11ee-bb0a-5bd84ee8291e, expected > 45474389658. Ignoring.

2023-07-10  2:20:50 0 [Note] WSREP: Member 1(fx114_db1) responds to vote on e4e16f9c-8779-11eb-b356-c3718f5f11e0:45474182196,0000000000000000: Success

2023-07-10  2:20:50 0 [Warning] WSREP: Received bogus VOTE message: 45474182196.0, from node cc3e6460-1ea6-11ee-88f2-4a01f1fdd917, expected > 45474389637. Ignoring.

2023-07-10  2:24:49 0 [Note] /usr/local/libexec/mariadbd (initiated by: unknown): Normal shutdown

2023-07-10  2:24:49 0 [Note] WSREP: Shutdown replication

2023-07-10  2:24:49 0 [Note] WSREP: Server status change joined -> disconnecting

An user noted that the following command didn't work in that state and Graceful shutdown executed on each node have no effect. Only possible way to stop the nodes is with kill -9.

SET GLOBAL wsrep_provider_options='pc.bootstrap=YES' ;

SET GLOBAL wsrep_provider_options="evs.evict=1de97dad-f609-11e5-8a50-ce2e621b0c42";

Attachments

Issue Links

relates to

MDEV-32122 Node killing could cause applier to fail on write rows event leading inconsistency

Closed

MDEV-33939 Assertion failure on lock_rec_queue_validate on multi-master load

Stalled

Activity

Ascending order - Click to sort in descending order

Allen Lee (Inactive) created issue - 2023-07-10 22:54

Allen Lee (Inactive) made changes - 2023-07-10 23:52

Field	Original Value	New Value
Summary	eadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos xxx, Internal MariaDB error code: 1213	Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos xxx, Internal MariaDB error code: 1213

Julien Fritsch made changes - 2023-07-11 06:57

Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]

Alexey made changes - 2023-07-12 09:33

Assignee

Alexey [ yurchenko ]

Alexey made changes - 2023-07-12 09:33

Status

Open [ 1 ]

In Progress [ 3 ]

Alexey added a comment - 2023-07-14 15:10

allen.lee@mariadb.com,

Could you please check if there are any tables which might be referencing db.tablex through foreign keys?

Alexey added a comment - 2023-07-14 15:10 allen.lee@mariadb.com , Could you please check if there are any tables which might be referencing db.tablex through foreign keys?

Alexey made changes - 2023-07-21 10:33

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Julien Fritsch made changes - 2023-07-31 10:34

Status

Stalled [ 10000 ]

Needs Feedback [ 10501 ]

Julien Fritsch made changes - 2023-07-31 10:34

Priority

Major [ 3 ]

Critical [ 2 ]

Julien Fritsch made changes - 2023-09-04 10:17

Fix Version/s		N/A [ 14700 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Incomplete [ 4 ]
Status	Needs Feedback [ 10501 ]	Closed [ 6 ]

Juan made changes - 2023-09-13 17:11

Resolution	Incomplete [ 4 ]
Status	Closed [ 6 ]	Stalled [ 10000 ]

Richard Stracke made changes - 2023-09-28 05:05

Link

This issue relates to ~~MDEV-32122~~ [ ~~MDEV-32122~~ ]

Julien Fritsch made changes - 2023-10-19 11:13

Status

Stalled [ 10000 ]

Needs Feedback [ 10501 ]

Yakov Kushnirsky made changes - 2023-10-19 22:58

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Julien Fritsch made changes - 2023-10-20 06:05

Assignee

Alexey [ yurchenko ]

Yakov Kushnirsky [ JIRAUSER48657 ]

Julien Fritsch made changes - 2023-10-20 06:05

Status

Open [ 1 ]

Needs Feedback [ 10501 ]

Yakov Kushnirsky added a comment - 2023-10-20 17:47

posted DDL, variables and logs to MENT. Will delete and close when no longer needed.

Yakov Kushnirsky added a comment - 2023-10-20 17:47 posted DDL, variables and logs to MENT. Will delete and close when no longer needed.

Yakov Kushnirsky made changes - 2023-10-20 17:47

Status

Needs Feedback [ 10501 ]

Open [ 1 ]

Yakov Kushnirsky made changes - 2023-11-07 15:25

Assignee

Yakov Kushnirsky [ JIRAUSER48657 ]

Alexey [ yurchenko ]

Julien Fritsch made changes - 2024-02-28 14:39

Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]
Fix Version/s	N/A [ 14700 ]

Julien Fritsch made changes - 2024-02-28 14:40

Link

This issue duplicates MENT-1967 [ MENT-1967 ]

Julien Fritsch made changes - 2024-02-28 14:43

Link

This issue is blocked by MENT-1967 [ MENT-1967 ]

Jan Lindström made changes - 2024-05-13 06:01

Assignee

Alexey [ yurchenko ]

Jan Lindström [ JIRAUSER53125 ]

Jan Lindström made changes - 2024-05-21 12:48

Status

Open [ 1 ]

In Progress [ 3 ]

Jan Lindström made changes - 2024-05-21 12:49

Assignee	Jan Lindström [ JIRAUSER53125 ]	Seppo Jaakola [ seppo ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström made changes - 2024-06-07 06:12

Assignee

Seppo Jaakola [ seppo ]

Jan Lindström [ JIRAUSER53125 ]

Jan Lindström made changes - 2024-06-07 06:12

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström added a comment - 2024-06-07 06:12

https://github.com/MariaDB/server/pull/3315

Jan Lindström added a comment - 2024-06-07 06:12 https://github.com/MariaDB/server/pull/3315

Jan Lindström made changes - 2024-06-07 06:12

Assignee	Jan Lindström [ JIRAUSER53125 ]	Marko Mäkelä [ marko ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström added a comment - 2024-06-10 07:38

marko Review comments addressed on https://github.com/MariaDB/server/commit/9327ab46864a17b8bc66bd324a025c3d6b5292c2

Jan Lindström added a comment - 2024-06-10 07:38 marko Review comments addressed on https://github.com/MariaDB/server/commit/9327ab46864a17b8bc66bd324a025c3d6b5292c2

Jan Lindström made changes - 2024-06-10 07:40

Summary

Deadlock found when trying to get lock; try restarting transaction, Error_code: 1213; handler error HA_ERR_LOCK_DEADLOCK; the event's master log FIRST, end_log_pos xxx, Internal MariaDB error code: 1213

Deadlock found when trying to get lock during applying

Marko Mäkelä made changes - 2024-06-13 05:31

Assignee	Marko Mäkelä [ marko ]	Jan Lindström [ JIRAUSER53125 ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Jan Lindström added a comment - 2024-06-13 06:58

https://github.com/MariaDB/server/pull/3315

Jan Lindström added a comment - 2024-06-13 06:58 https://github.com/MariaDB/server/pull/3315

Jan Lindström made changes - 2024-06-13 06:58

Assignee	Jan Lindström [ JIRAUSER53125 ]	Julius Goryavsky [ sysprg ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström made changes - 2024-06-17 10:27

Link

This issue relates to MDEV-33939 [ MDEV-33939 ]

Julius Goryavsky made changes - 2024-06-19 13:32

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Julius Goryavsky added a comment - 2024-06-19 13:35

Fix merged with head revision: https://github.com/MariaDB/server/commit/ee974ca5e0f00e928873179c5dfc2f040ab5aefd

Julius Goryavsky added a comment - 2024-06-19 13:35 Fix merged with head revision: https://github.com/MariaDB/server/commit/ee974ca5e0f00e928873179c5dfc2f040ab5aefd

Julius Goryavsky made changes - 2024-06-19 13:35

issue.field.resolutiondate

2024-06-19 13:35:25.0

2024-06-19 13:35:24.838

Julius Goryavsky made changes - 2024-06-19 13:35

Fix Version/s		10.6.19 [ 29833 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Resolution		Fixed [ 1 ]
Status	Stalled [ 10000 ]	Closed [ 6 ]

JiraAutomate made changes - 2024-06-19 13:35

Fix Version/s		10.11.9 [ 29834 ]
Fix Version/s		11.1.6 [ 29835 ]
Fix Version/s		11.2.5 [ 29836 ]
Fix Version/s		11.4.3 [ 29837 ]

Jira Automation (IT) made changes - 2024-07-04 00:58

Zendesk Related Tickets		202047 202128 201581
Zendesk active tickets		201581

Jira Automation (IT) made changes - 2024-08-14 04:59

Zendesk active tickets

201581

People

Assignee:: Julius Goryavsky

Reporter:: Allen Lee (Inactive)

Votes:: 1 Vote for this issue

Watchers:: 13 Start watching this issue

Dates

Created:: 2023-07-10 22:54

Updated:: 2024-08-14 04:59

Resolved:: 2024-06-19 13:35

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration