Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Incomplete
-
10.5.13, 10.5
-
None
Description
We've seen one node unable to join back to cluster because of error
2021/12/06 10:19:09 Peer list updated |
was []
|
now [mysql-0.mysql.default.svc.cluster.local mysql-1.mysql.default.svc.cluster.local mysql-2.mysql.default.svc.cluster.local] |
2021/12/06 10:19:09 execing: /opt/galera/on-start.sh with stdin: mysql-0.mysql.default.svc.cluster.local |
mysql-1.mysql.default.svc.cluster.local |
mysql-2.mysql.default.svc.cluster.local |
2021/12/06 10:19:09 *** [Galera] Joining cluster: mysql-1.mysql.default.svc.cluster.local,mysql-2.mysql.default.svc.cluster.local |
2021/12/06 10:19:10 Peer finder exiting |
Galera - Determining recovery position...
|
galera-recovery.sh: Attempting to recover GTID positon...
|
2021-12-06 10:19:11 0 [Note] mysqld (mysqld 10.5.13-MariaDB-1:10.5.13+maria~focal) starting as process 49 ... |
galera-recovery.sh: Found WSREP position: 27703fb6-fab1-11eb-916a-3f7c89091ca1:47824716 |
Galera recovery position: --wsrep_start_position=27703fb6-fab1-11eb-916a-3f7c89091ca1:47824716 |
2021-12-06 10:19:12 0 [Note] mysqld (mysqld 10.5.13-MariaDB-1:10.5.13+maria~focal) starting as process 1 ... |
2021-12-06 10:19:12 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 27703fb6-fab1-11eb-916a-3f7c89091ca1:47824716 |
2021-12-06 10:19:12 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so' |
2021-12-06 10:19:12 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully. |
2021-12-06 10:19:12 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. |
2021-12-06 10:19:12 0 [Note] WSREP: Found saved state: 27703fb6-fab1-11eb-916a-3f7c89091ca1:-1, safe_to_bootstrap: 1 |
2021-12-06 10:19:12 0 [Note] WSREP: GCache DEBUG: opened preamble: |
Version: 2 |
UUID: 27703fb6-fab1-11eb-916a-3f7c89091ca1
|
Seqno: -1 - -1 |
Offset: -1 |
Synced: 0 |
2021-12-06 10:19:12 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 27703fb6-fab1-11eb-916a-3f7c89091ca1, offset: -1 |
2021-12-06 10:19:12 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete. |
2021-12-06 10:19:12 0 [Warning] WSREP: Exception while mapping writeset 7493998576037527552 into [47818165, 47824717): 'deque::_M_new_elements_at_back'. Aborting GCache recovery. |
2021-12-06 10:19:12 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete. |
2021-12-06 10:19:12 0 [Note] WSREP: Recovering GCache ring buffer: found gapless sequence 47818165-47824716 |
2021-12-06 10:19:12 0 [Note] WSREP: GCache::RingBuffer unused buffers scan... 0.0% ( 0/133983928 bytes) complete. |
211206 10:19:12 [ERROR] mysqld got signal 11 ; |
This could be because you hit a bug. It is also possible that this binary |
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
We will try our best to scrape up some info that will hopefully help |
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail. |
Server version: 10.5.13-MariaDB-1:10.5.13+maria~focal |
key_buffer_size=0 |
read_buffer_size=131072 |
max_used_connections=0 |
max_threads=252 |
thread_count=0 |
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 554746 K bytes of memory |
Hope that's ok; if not, decrease some variables in the equation. |
Thread pointer: 0x0 |
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went |
terribly wrong...
|
stack_bottom = 0x0 thread_stack 0x49000 |
mysqld(my_print_stacktrace+0x32)[0x564cb13b5e42] |
Printing to addr2line failed
|
mysqld(handle_fatal_signal+0x485)[0x564cb0e059a5] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fe5dc4463c0] |
/usr/lib/galera/libgalera_smm.so(+0x1b553c)[0x7fe5d65e653c] |
/usr/lib/galera/libgalera_smm.so(+0x1b7de7)[0x7fe5d65e8de7] |
/usr/lib/galera/libgalera_smm.so(+0x1b9362)[0x7fe5d65ea362] |
/usr/lib/galera/libgalera_smm.so(+0x1bd727)[0x7fe5d65ee727] |
/usr/lib/galera/libgalera_smm.so(+0x7b6b8)[0x7fe5d64ac6b8] |
/usr/lib/galera/libgalera_smm.so(+0x4ee82)[0x7fe5d647fe82] |
mysqld(_ZN5wsrep18wsrep_provider_v26C1ERNS_12server_stateERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_RKNS_8provider8servicesE+0x1dc)[0x564cb144e02c] |
mysqld(_ZN5wsrep8provider13make_providerERNS_12server_stateERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESA_RKNS0_8servicesE+0x54)[0x564cb144b034] |
mysqld(_ZN5wsrep12server_state13load_providerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_RKNS_8provider8servicesE+0x1f3)[0x564cb14363f3] |
mysqld(_Z10wsrep_initv+0x233)[0x564cb10cf2f3] |
mysqld(_Z18wsrep_init_startupb+0x16)[0x564cb10cf8e6] |
mysqld(+0x6983f9)[0x564cb0b1e3f9] |
mysqld(_Z11mysqld_mainiPPc+0x401)[0x564cb0b23201] |
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fe5dbf2d0b3] |
mysqld(_start+0x2e)[0x564cb0b17ede] |
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains |
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes |
Max core file size unlimited unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes unlimited unlimited processes
|
Max open files 1048576 1048576 files |
Max locked memory 67108864 67108864 bytes |
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 128321 128321 signals |
Max msgqueue size 819200 819200 bytes |
Max nice priority 0 0 |
Max realtime priority 0 0 |
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/share/apport/apport %p %s %c %d %P %E
|
The workaround was to let node fo the full resync from other 2 nodes
ENV: 3 master setup
Config:
data:
|
galera.cnf: |
|
[galera]
|
user = mysql
|
bind-address = 0.0.0.0 |
default_storage_engine = InnoDB
|
binlog_format = ROW
|
innodb_autoinc_lock_mode = 2 |
innodb_flush_log_at_trx_commit = 0 |
query_cache_size = 0 |
query_cache_type = 0 |
binlog_cache_size = 61440 |
|
# MariaDB Galera settings
|
wsrep_on=ON
|
wsrep_provider=/usr/lib/galera/libgalera_smm.so
|
wsrep_sst_method=rsync
|
wsrep_slave_threads=8 |
wsrep_sync_wait=7 |
|
# Cluster settings (automatically updated)
|
wsrep_cluster_address=gcomm:// |
wsrep_cluster_name=mysql
|
wsrep_node_address=127.0.0.1 |
mariadb.cnf: "[client]\ndefault-character-set = utf8\n[mysqld]\ncore-file\nunix_socket
|
= OFF\nperformance_schema = ON\ncharacter-set-server = utf8\ncollation-server
|
= utf8_general_ci\nignore-db-dirs = lost+found \nmax_connections = 250\ninteractive_timeout |
= 450 \nwait_timeout = 450\ntable_definition_cache = 2100\n# InnoDB tuning\ninnodb_buffer_pool_size |
= 8400MB\ninnodb_log_file_size = 2G\n"
|