[MDEV-25114] Crash: WSREP: invalid state ROLLED_BACK (FATAL) - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Blocker
Resolution: Fixed
Affects Version/s: 10.3.28, 10.2(EOL), 10.4(EOL), 10.5
Fix Version/s: 10.2.41, 10.3.32, 10.4.22, 10.5.13
Component/s: Galera
Labels:
- crash
- hang
- not-10.6
- not-10.7
- regression
Environment:

Hide
CentOS 7.9.2009

mysqld would have been started with the following arguments:
--datadir=/data/mariadb --socket=/var/lib/mysql/mysql.sock --user=mysql --symbolic-links=0 --max_allowed_packet=16M --thread_cache_size=8 --max_connections=1500 --slow_query_log=1 --log_error=/data/mariadb/mariadb.log --innodb_buffer_pool_size=49152M --innodb_log_file_size=2048M --innodb_log_buffer_size=16M --innodb_print_all_deadlocks=on --log-warnings=2 --plugin_load_add=auth_socket --default_storage_engine=InnoDB --innodb_file_per_table=1 --innodb_flush_log_at_trx_commit=0 --innodb_doublewrite=1 --log_slave_updates=1 --log_bin=bin-log --server_id=6666 --binlog_format=ROW --innodb_autoinc_lock_mode=2 --expire_logs_days=1 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm://[redacted] --wsrep_node_address=[redacted] --wsrep_sst_method=mariabackup --wsrep_sst_auth=[redacted] --wsrep_provider_options=evs.keepalive_period = PT1S; evs.inactive_check_period = PT1S; evs.suspect_timeout = PT5S; evs.inactive_timeout = PT15S; evs.install_timeout = PT15S; gcache.size=2G --wsrep_on=ON --wsrep_log_conflicts=1

Show
CentOS 7.9.2009 mysqld would have been started with the following arguments: --datadir=/data/mariadb --socket=/var/lib/mysql/mysql.sock --user=mysql --symbolic-links=0 --max_allowed_packet=16M --thread_cache_size=8 --max_connections=1500 --slow_query_log=1 --log_error=/data/mariadb/mariadb.log --innodb_buffer_pool_size=49152M --innodb_log_file_size=2048M --innodb_log_buffer_size=16M --innodb_print_all_deadlocks=on --log-warnings=2 --plugin_load_add=auth_socket --default_storage_engine=InnoDB --innodb_file_per_table=1 --innodb_flush_log_at_trx_commit=0 --innodb_doublewrite=1 --log_slave_updates=1 --log_bin=bin-log --server_id=6666 --binlog_format=ROW --innodb_autoinc_lock_mode=2 --expire_logs_days=1 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm://[redacted] --wsrep_node_address=[redacted] --wsrep_sst_method=mariabackup --wsrep_sst_auth=[redacted] --wsrep_provider_options=evs.keepalive_period = PT1S; evs.inactive_check_period = PT1S; evs.suspect_timeout = PT5S; evs.inactive_timeout = PT15S; evs.install_timeout = PT15S; gcache.size=2G --wsrep_on=ON --wsrep_log_conflicts=1

Description

About 29 hours after updating a previously very stable 3 server MariaDB Galera cluster from 10.3.27 to 10.3.28, one of the nodes crashed with the following message:

2021-03-10 18:22:48 0 [ERROR] WSREP: invalid state ROLLED_BACK (FATAL)

         at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:abort_trx():735

2021-03-10 18:22:48 0 [ERROR] WSREP: cancel commit bad exit: 7 33792346039

210310 18:22:48 [ERROR] mysqld got signal 6 ;

Attached is a bit more log (coincidentally had been running with conflict logging enabled), but I don't think there's much more I can provide right now. I'm still logging this since before the update to 10.3.28 the cluster had been running very stable for months, and this could be somehow related to ~~MDEV-25111~~, which we also encountered first time right after updating to 10.3.28.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

gdb.txt
207 kB
2021-06-02 06:05
mariadb-invalid-state-crash.txt
5 kB
2021-03-11 07:44

Issue Links

blocks

MDEV-25368 Galera cluster hangs on Freeing items

Closed

causes

MDEV-24294 MariaDB - Cluster freezes if node hangs

Closed

MDEV-25992 Galera 3 Server crash with signal 6 after RBR event apply failed

Closed

MDEV-26099 MariaDB 10.5.10/10.5.11 Galera assertion crash

Closed

is caused by

MDEV-23328 Server hang due to Galera lock conflict resolution

Closed

is duplicated by

MDEV-28604 MariaDB crashing very often

Closed

relates to

MDEV-24915 Galera conflict resolution is unnecessarily complex

Closed

MDEV-25410 Assertion `state_ == s_exec' failed - mysqld got signal 6

Closed

MDEV-25609 Signal 11 on wsrep_mysqld.cc:2620

Closed

MDEV-24397 MariaDB Galera Server Crashes on Large DELETE

Closed

MDEV-25111 Long semaphore wait (> 800 secs), server stops responding

Closed

(1 is duplicated by, 5 relates to)

Activity

Ascending order - Click to sort in descending order

Ere Maijala created issue - 2021-03-11 07:48

Ere Maijala made changes - 2021-03-11 07:48

Field	Original Value	New Value
Link		This issue relates to ~~MDEV-25111~~ [ ~~MDEV-25111~~ ]

Ere Maijala made changes - 2021-03-11 08:12

Environment

CentOS 7.9.2009

mysqld would have been started with the following arguments:
--datadir=/data/mariadb --socket=/var/lib/mysql/mysql.sock --user=mysql --symbolic-links=0 --max_allowed_packet=16M --thread_cache_size=8 --max_connections=1500 --slow_query_log=1 --log_error=/data/mariadb/mariadb.log --innodb_buffer_pool_size=49152M --innodb_log_file_size=2048M --innodb_log_buffer_size=16M --innodb_print_all_deadlocks=on --log-warnings=2 --plugin_load_add=auth_socket --default_storage_engine=InnoDB --innodb_file_per_table=1 --innodb_flush_log_at_trx_commit=0 --innodb_doublewrite=1 --log_slave_updates=1 --log_bin=bin-log --server_id=6666 --binlog_format=ROW --innodb_autoinc_lock_mode=2 --expire_logs_days=1 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm://finna-fe-db-1.csc.fi,finna-fe-db-2.csc.fi,finna-fe-db-3.csc.fi --wsrep_node_address=finna-fe-db-3.csc.fi --wsrep_sst_method=mariabackup --wsrep_sst_auth=mariabackup:aK92gTx --wsrep_provider_options=evs.keepalive_period = PT1S; evs.inactive_check_period = PT1S; evs.suspect_timeout = PT5S; evs.inactive_timeout = PT15S; evs.install_timeout = PT15S; gcache.size=2G --wsrep_on=ON --wsrep_log_conflicts=1

CentOS 7.9.2009

mysqld would have been started with the following arguments:
--datadir=/data/mariadb --socket=/var/lib/mysql/mysql.sock --user=mysql --symbolic-links=0 --max_allowed_packet=16M --thread_cache_size=8 --max_connections=1500 --slow_query_log=1 --log_error=/data/mariadb/mariadb.log --innodb_buffer_pool_size=49152M --innodb_log_file_size=2048M --innodb_log_buffer_size=16M --innodb_print_all_deadlocks=on --log-warnings=2 --plugin_load_add=auth_socket --default_storage_engine=InnoDB --innodb_file_per_table=1 --innodb_flush_log_at_trx_commit=0 --innodb_doublewrite=1 --log_slave_updates=1 --log_bin=bin-log --server_id=6666 --binlog_format=ROW --innodb_autoinc_lock_mode=2 --expire_logs_days=1 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_cluster_address=gcomm://[redacted] --wsrep_node_address=[redacted] --wsrep_sst_method=mariabackup --wsrep_sst_auth=[redacted] --wsrep_provider_options=evs.keepalive_period = PT1S; evs.inactive_check_period = PT1S; evs.suspect_timeout = PT5S; evs.inactive_timeout = PT15S; evs.install_timeout = PT15S; gcache.size=2G --wsrep_on=ON --wsrep_log_conflicts=1

Valerii Kravchuk made changes - 2021-03-19 07:39

Labels		regression
Priority	Major [ 3 ]	Critical [ 2 ]

Jan Lindström (Inactive) made changes - 2021-03-19 09:44

Assignee

Seppo Jaakola [ seppo ]

Julien Fritsch made changes - 2021-03-26 07:32

Fix Version/s

10.3 [ 22126 ]

Seppo Jaakola made changes - 2021-04-21 10:08

Status

Open [ 1 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-05-26 10:58

Assignee

Seppo Jaakola [ seppo ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-06-01 06:49

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-06-01 06:49

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Jan Lindström (Inactive) made changes - 2021-06-02 06:05

Attachment

gdb.txt [ 57937 ]

Julien Fritsch made changes - 2021-06-03 09:22

Link

This issue relates to ~~MDEV-25410~~ [ ~~MDEV-25410~~ ]

Julien Fritsch made changes - 2021-06-03 09:22

Link

This issue relates to ~~MDEV-25835~~ [ ~~MDEV-25835~~ ]

Julien Fritsch made changes - 2021-06-03 09:22

Link

This issue relates to ~~MDEV-25609~~ [ ~~MDEV-25609~~ ]

Seppo Jaakola made changes - 2021-06-07 08:56

Assignee	Seppo Jaakola [ seppo ]	Jan Lindström [ jplindst ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2021-06-15 10:09

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-06-16 08:09

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-06-17 08:32

Assignee	Jan Lindström [ jplindst ]	Marko Mäkelä [ marko ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Marko Mäkelä made changes - 2021-06-17 14:04

Assignee	Marko Mäkelä [ marko ]	Jan Lindström [ jplindst ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:45

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	Stalled [ 10000 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:46

Priority

Critical [ 2 ]

Blocker [ 1 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:46

Affects Version/s

10.2 [ 14601 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:46

Affects Version/s

10.4 [ 22408 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:46

Affects Version/s

10.5 [ 23123 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:46

Fix Version/s		10.2.39 [ 25731 ]
Fix Version/s		10.3.30 [ 25732 ]
Fix Version/s		10.4.20 [ 25733 ]
Fix Version/s		10.5.11 [ 25734 ]
Fix Version/s	10.3 [ 22126 ]

Jan Lindström (Inactive) made changes - 2021-06-18 06:47

Link

This issue is part of MENT-1202 [ MENT-1202 ]

Sergei Golubchik made changes - 2021-06-22 08:34

Fix Version/s		10.2 [ 14601 ]
Fix Version/s		10.3 [ 22126 ]
Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s	10.2.39 [ 25731 ]
Fix Version/s	10.3.30 [ 25732 ]
Fix Version/s	10.4.20 [ 25733 ]
Fix Version/s	10.5.11 [ 25734 ]

Jan Lindström (Inactive) made changes - 2021-06-24 07:40

Link

This issue is part of MENT-1202 [ MENT-1202 ]

Jan Lindström (Inactive) made changes - 2021-06-24 07:41

Link

This issue blocks MENT-1202 [ MENT-1202 ]

Jan Lindström (Inactive) made changes - 2021-06-24 08:32

Link

This issue blocks TODO-2984 [ TODO-2984 ]

Seppo Jaakola made changes - 2021-06-30 08:52

Link

This issue relates to ~~MDEV-25368~~ [ ~~MDEV-25368~~ ]

Jan Lindström (Inactive) made changes - 2021-07-01 10:51

Link

This issue causes ~~MDEV-25992~~ [ ~~MDEV-25992~~ ]

Sergei Golubchik made changes - 2021-07-09 12:16

Description

About 29 hours after updating a previously very stable 3 server MariaDB Galera cluster from 10.3.27 to 10.3.28, one of the nodes crashed with the following message:

2021-03-10 18:22:48 0 [ERROR] WSREP: invalid state ROLLED_BACK (FATAL)
at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:abort_trx():735
2021-03-10 18:22:48 0 [ERROR] WSREP: cancel commit bad exit: 7 33792346039
210310 18:22:48 [ERROR] mysqld got signal 6 ;

Attached is a bit more log (coincidentally had been running with conflict logging enabled), but I don't think there's much more I can provide right now. I'm still logging this since before the update to 10.3.28 the cluster had been running very stable for months, and this could be somehow related to ~~MDEV-25111~~, which we also encountered first time right after updating to 10.3.28.

About 29 hours after updating a previously very stable 3 server MariaDB Galera cluster from 10.3.27 to 10.3.28, one of the nodes crashed with the following message:
{noformat}
2021-03-10 18:22:48 0 [ERROR] WSREP: invalid state ROLLED_BACK (FATAL)
at /home/buildbot/buildbot/build/galera/src/replicator_smm.cpp:abort_trx():735
2021-03-10 18:22:48 0 [ERROR] WSREP: cancel commit bad exit: 7 33792346039
210310 18:22:48 [ERROR] mysqld got signal 6 ;
{noformat}
Attached is a bit more log (coincidentally had been running with conflict logging enabled), but I don't think there's much more I can provide right now. I'm still logging this since before the update to 10.3.28 the cluster had been running very stable for months, and this could be somehow related to ~~MDEV-25111~~, which we also encountered first time right after updating to 10.3.28.

Sergei Golubchik made changes - 2021-07-28 17:35

Assignee	Sergei Golubchik [ serg ]	Jan Lindström [ jplindst ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-07-29 04:26

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Jan Lindström (Inactive) made changes - 2021-07-29 06:30

Assignee

Seppo Jaakola [ seppo ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-07-29 06:31

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-07-29 06:31

Labels

regression

crash regression

Jan Lindström (Inactive) made changes - 2021-07-29 06:31

Labels

crash regression

crash hang regression

Sergei Golubchik made changes - 2021-07-30 10:13

Link

This issue blocks MENT-1284 [ MENT-1284 ]

Sergei Golubchik made changes - 2021-07-30 10:15

Priority

Blocker [ 1 ]

Critical [ 2 ]

Jan Lindström (Inactive) made changes - 2021-08-03 11:45

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Ralf Gebhardt made changes - 2021-08-09 13:12

Priority

Critical [ 2 ]

Blocker [ 1 ]

Jan Lindström (Inactive) made changes - 2021-08-12 08:32

Labels

crash hang regression

crash hang need_feedback regression

Jan Lindström (Inactive) made changes - 2021-08-12 08:32

Assignee

Jan Lindström [ jplindst ]

Sergei Golubchik [ serg ]

Jan Lindström (Inactive) made changes - 2021-08-12 10:57

Labels

crash hang need_feedback regression

crash hang regression

Julien Fritsch made changes - 2021-08-12 12:08

Link

This issue blocks ~~MDEV-25835~~ [ ~~MDEV-25835~~ ]

Julien Fritsch made changes - 2021-08-12 12:08

Link

This issue relates to ~~MDEV-25835~~ [ ~~MDEV-25835~~ ]

Julien Fritsch made changes - 2021-08-16 09:49

Link

This issue blocks ~~MDEV-25368~~ [ ~~MDEV-25368~~ ]

Julien Fritsch made changes - 2021-08-16 09:49

Link

This issue relates to ~~MDEV-25368~~ [ ~~MDEV-25368~~ ]

Jan Lindström (Inactive) made changes - 2021-08-16 10:22

Assignee

Sergei Golubchik [ serg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-08-16 10:22

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-08-17 06:00

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2021-08-18 07:42

Link

This issue causes ~~MDEV-24294~~ [ ~~MDEV-24294~~ ]

Jan Lindström (Inactive) made changes - 2021-08-19 06:06

Assignee

Sergei Golubchik [ serg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-08-19 06:06

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-08-30 07:59

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-09-01 06:56

Status

In Progress [ 3 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-09-06 05:50

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Ramesh Sivaraman made changes - 2021-09-07 15:43

Assignee

Jan Lindström [ jplindst ]

Ramesh Sivaraman [ JIRAUSER48189 ]

Ramesh Sivaraman made changes - 2021-09-09 11:28

Assignee

Ramesh Sivaraman [ JIRAUSER48189 ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-09-09 11:31

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2021-09-13 06:49

Assignee

Sergei Golubchik [ serg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-09-13 06:49

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-09-13 06:49

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-09-16 04:36

Link

This issue causes ~~MDEV-26099~~ [ ~~MDEV-26099~~ ]

Jan Lindström (Inactive) made changes - 2021-09-16 12:22

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Seppo Jaakola made changes - 2021-09-22 06:48

Link

This issue is caused by ~~MDEV-23328~~ [ ~~MDEV-23328~~ ]

Jan Lindström (Inactive) made changes - 2021-10-21 12:54

Assignee

Sergei Golubchik [ serg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-10-21 12:54

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-10-21 12:55

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-10-21 15:04

Assignee	Jan Lindström [ jplindst ]	Sergei Golubchik [ serg ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) made changes - 2021-10-27 08:17

Link

This issue blocks ~~MDEV-25835~~ [ ~~MDEV-25835~~ ]

Jan Lindström (Inactive) made changes - 2021-10-27 10:06

Labels

crash hang regression

crash hang not-10.6 not-10.7 regression

Julien Fritsch made changes - 2021-10-29 08:52

Link

This issue blocks TODO-3199 [ TODO-3199 ]

Jan Lindström (Inactive) made changes - 2021-10-29 09:32

Assignee

Sergei Golubchik [ serg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-10-29 09:32

Status

In Review [ 10002 ]

Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-10-29 09:32

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2021-10-30 06:28

issue.field.resolutiondate

2021-10-30 06:28:15.0

2021-10-30 06:28:15.345

Jan Lindström (Inactive) made changes - 2021-10-30 06:28

Fix Version/s		10.2.41 [ 26032 ]
Fix Version/s		10.3.32 [ 26029 ]
Fix Version/s		10.4.22 [ 26031 ]
Fix Version/s		10.5.13 [ 26026 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Marko Mäkelä made changes - 2021-12-03 09:39

Link

This issue relates to ~~MDEV-24915~~ [ ~~MDEV-24915~~ ]

Sergei Golubchik made changes - 2021-12-06 21:53

Workflow

MariaDB v3 [ 120009 ]

MariaDB v4 [ 159016 ]

Roel Van de Paar made changes - 2022-03-16 06:50

Link

This issue relates to ~~MDEV-24397~~ [ ~~MDEV-24397~~ ]

Daniel Black made changes - 2022-05-18 08:43

Link

This issue is duplicated by ~~MDEV-28604~~ [ ~~MDEV-28604~~ ]

Jira Automation (IT) made changes - 2024-07-04 02:34

Zendesk Related Tickets

116083 144997 139390 171032 120022

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Ere Maijala

Votes:: 16 Vote for this issue

Watchers:: 37 Start watching this issue

Dates

Created:: 2021-03-11 07:48

Updated:: 2024-07-07 21:18

Resolved:: 2021-10-30 06:28

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration