Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.2.12, 10.2.30
Description
Due to a bug in interaction between native slave thread processing and MariaDB wsrep patch we may end up with crashes like these:
...
|
2020-01-09 11:16:38 140078043080448 [Note] WSREP: Member 2.0 (hss6geo246o) synced with group.
|
2020-01-09 12:18:41 140076876134144 [ERROR] Master 'to_master_0': Slave SQL: Node has dropped from cluster, Gtid 200-200-3694, Internal MariaDB error code: 1047
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': Slave SQL thread exiting, replication stopped in log 'binlog.000004' at position 10915199; GTID position '100-100-1123,200-200-3693'
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: Slave error due to node temporarily non-primarySQL slave will continue
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: slave restart: 3
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: ready state reached
|
...
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: Slave error due to node temporarily non-primarySQL slave will continue
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: slave restart: 3
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: ready state reached
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': Slave SQL thread initialized, starting replication in log 'binlog.000004' at position 10991571, relay log '/var/lib/mybinlog/relaylog-to_master_0.000006' position: 10407768; GTID position '100-100-1123,200-200-3834'
|
2020-01-09 12:18:41 140078447146752 [ERROR] Master 'to_master_0': WSREP: FSM: no such a transition REPLICATING -> ROLLED_BACK
|
200109 12:18:41 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.2.30-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=41
|
max_threads=153
|
thread_count=86
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f66686d5cd8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f66307161a0 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e16de2b47e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x55e16d8b0abd]
|
sigaction.c:0(__restore_rt)[0x7f668de916d0]
|
:0(__GI_raise)[0x7f668c394277]
|
:0(__GI_abort)[0x7f668c395968]
|
src/fsm.hpp:104(galera::FSM<galera::TrxHandle::State, galera::TrxHandle::Transition, galera::EmptyGuard, galera::EmptyAction>::shift_to(galera::TrxHandle::State))[0x7f6684fa61dc]
|
:0(galera::ReplicatorSMM::post_rollback(galera::TrxHandle*))[0x7f6684f9c6a6]
|
src/gu_mutex.hpp:38(gu::Mutex::unlock() const)[0x7f6684fad385]
|
/usr/sbin/mysqld(+0x6148b5)[0x55e16d8408b5]
|
/usr/sbin/mysqld(+0x6149e8)[0x55e16d8409e8]
|
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xe3)[0x55e16d8b3a73]
|
/usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x49)[0x55e16d7f80f9]
|
/usr/sbin/mysqld(_Z21wsrep_client_rollbackP3THD+0x60)[0x55e16d856810]
|
/usr/sbin/mysqld(+0x62acab)[0x55e16d856cab]
|
/usr/sbin/mysqld(start_wsrep_THD+0x4fb)[0x55e16d84575b]
|
pthread_create.c:0(start_thread)[0x7f668de89e25]
|
/lib64/libc.so.6(clone+0x6d)[0x7f668c45cbad]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 2571
|
Status: NOT_KILLED
|
...
|
or:
...
|
2020-01-09 12:41:56 139923526936320 [Note] Master 'to_master_0': Slave SQL thread exiting, replication stopped in log 'binlog.000004' at position 76754168; GTID position '100-100-29316,200-200-123484'
|
2020-01-09 12:41:56 139923526936320 [Note] Master 'to_master_0': WSREP: Slave error due to node temporarily non-primarySQL slave will continue
|
2020-01-09 12:41:56 139923526936320 [Note] Master 'to_master_0': WSREP: slave restart: 3
|
2020-01-09 12:41:56 139923526936320 [Note] Master 'to_master_0': WSREP: ready state reached
|
2020-01-09 12:41:56 139923526936320 [Note] Master 'to_master_0': Slave SQL thread initialized, starting replication in log 'binlog.000004' at position 76754168, relay log '/var/lib/mybinlog/relaylog-to_master_0.000002' position: 17291006; GTID position '100-100-29316,200-200-123484'
|
200109 12:41:56 [ERROR] mysqld got signal 11 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.2.30-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=23
|
max_threads=153
|
thread_count=55
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f4260316ee8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f427c1e51a0 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562911c4c47e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x5629116d1abd]
|
sigaction.c:0(__restore_rt)[0x7f42cd70b6d0]
|
/usr/sbin/mysqld(_Z19wsrep_register_htonP3THDb+0x84)[0x562911661a84]
|
/usr/sbin/mysqld(_Z12trans_commitP3THD+0x10d)[0x562911618f6d]
|
/usr/sbin/mysqld(_ZN13Xid_log_event14do_apply_eventEP14rpl_group_info+0xca)[0x5629117c07ba]
|
/usr/sbin/mysqld(+0x46d9eb)[0x5629114ba9eb]
|
/usr/sbin/mysqld(handle_slave_sql+0x1533)[0x5629114c38b3]
|
pthread_create.c:0(start_thread)[0x7f42cd703e25]
|
/lib64/libc.so.6(clone+0x6d)[0x7f42cbcd6bad]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 1103
|
Status: NOT_KILLED
|
or even like this:
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: slave restart: 3
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': WSREP: ready state reached
|
2020-01-09 12:18:41 140076876134144 [Note] Master 'to_master_0': Slave SQL thread initialized, starting replication in log 'binlog.000004' at position 10991571, relay log '/var/lib/mybinlog/relaylog-to_master_0.000006' position: 10407768; GTID position '100-100-1123,200-200-3834'
|
2020-01-09 12:18:41 140078447146752 [ERROR] Master 'to_master_0': WSREP: FSM: no such a transition REPLICATING -> ROLLED_BACK
|
200109 12:18:41 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.2.30-MariaDB-log
|
key_buffer_size=134217728
|
read_buffer_size=131072
|
max_used_connections=41
|
max_threads=153
|
thread_count=86
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f66686d5cd8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f66307161a0 thread_stack 0x49000
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e16de2b47e]
|
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x55e16d8b0abd]
|
sigaction.c:0(__restore_rt)[0x7f668de916d0]
|
:0(__GI_raise)[0x7f668c394277]
|
:0(__GI_abort)[0x7f668c395968]
|
src/fsm.hpp:104(galera::FSM<galera::TrxHandle::State, galera::TrxHandle::Transition, galera::EmptyGuard, galera::EmptyAction>::shift_to(galera::TrxHandle::State))[0x7f6684fa61dc]
|
:0(galera::ReplicatorSMM::post_rollback(galera::TrxHandle*))[0x7f6684f9c6a6]
|
src/gu_mutex.hpp:38(gu::Mutex::unlock() const)[0x7f6684fad385]
|
/usr/sbin/mysqld(+0x6148b5)[0x55e16d8408b5]
|
/usr/sbin/mysqld(+0x6149e8)[0x55e16d8409e8]
|
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xe3)[0x55e16d8b3a73]
|
/usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x49)[0x55e16d7f80f9]
|
/usr/sbin/mysqld(_Z21wsrep_client_rollbackP3THD+0x60)[0x55e16d856810]
|
/usr/sbin/mysqld(+0x62acab)[0x55e16d856cab]
|
/usr/sbin/mysqld(start_wsrep_THD+0x4fb)[0x55e16d84575b]
|
pthread_create.c:0(start_thread)[0x7f668de89e25]
|
/lib64/libc.so.6(clone+0x6d)[0x7f668c45cbad]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 2571
|
Status: NOT_KILLED
|
Attachments
Issue Links
- relates to
-
-
- Closed
-