[MDEV-30768] MariaDB 10.3.38 Galera cluster nodes are crushing Created: 2023-03-01  Updated: 2023-06-06  Resolved: 2023-06-06

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.3.38
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Recep Direk Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None


 Description   

Hi,
I have 3 node multi master Galera cluster which is running on MariaDB 10.3.38 version.
After upgrading database from 10.3.24 we got these error and nodes got crashed.

Can you please help us to sort it out these issue?

2023-03-01 21:38:12 0 [Note] WSREP: Member 0.0 (mariadb1-u) requested state transfer from '*any*'. Selected 1.0 (mariadb4-u)(SYNCED) as donor.
2023-03-01 21:38:12 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5218154551)
2023-03-01 21:38:12 2 [Note] WSREP: Requesting state transfer: success, donor: 1
2023-03-01 21:38:12 2 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218154494
2023-03-01 21:38:12 0 [Note] WSREP: 1.0 (mariadb4-u): State transfer to 0.0 (mariadb1-u) complete.
2023-03-01 21:38:12 0 [Note] WSREP: Member 1.0 (mariadb4-u) synced with group.
2023-03-01 21:38:12 0 [Note] WSREP: SST complete, seqno: 5218150076
2023-03-01 21:38:12 0 [Note] Loaded 'ha_federated.so' with offset 0x7f71cd5b0000
2023-03-01 21:38:12 0 [Warning] Plugin 'FEDERATED' is of maturity level gamma while the server is stable
2023-03-01 21:38:12 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
2023-03-01 21:38:12 0 [Note] InnoDB: Using Linux native AIO
2023-03-01 21:38:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-03-01 21:38:12 0 [Note] InnoDB: Uses event mutexes
2023-03-01 21:38:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-03-01 21:38:12 0 [Note] InnoDB: Number of pools: 1
2023-03-01 21:38:12 0 [Note] InnoDB: Using SSE2 crc32 instructions
2023-03-01 21:38:12 0 [Note] InnoDB: Initializing buffer pool, total size = 48G, instances = 8, chunk size = 128M
2023-03-01 21:38:13 0 [Note] InnoDB: Completed initialization of buffer pool
2023-03-01 21:38:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-03-01 21:38:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2023-03-01 21:38:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-03-01 21:38:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-03-01 21:38:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-03-01 21:38:13 0 [Note] InnoDB: 10.3.38 started; log sequence number 18289586082679; transaction id 17367262174
2023-03-01 21:38:13 0 [Note] InnoDB: Loading buffer pool(s) from /srv/galera/mysql/ib_buffer_pool
2023-03-01 21:38:13 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-01 21:38:13 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-03-01 21:38:13 0 [Note] WSREP: Signalling provider to continue.
2023-03-01 21:38:13 0 [Note] WSREP: SST received: 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
2023-03-01 21:38:13 0 [Note] Reading of all Master_info entries succeeded
2023-03-01 21:38:13 0 [Note] Added new Master_info '' to hash table
2023-03-01 21:38:13 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2023-03-01 21:38:13 2 [Note] WSREP: Receiving IST: 4418 writesets, seqnos 5218150076-5218154494
2023-03-01 21:38:13 0 [Note] WSREP: Receiving IST...  0.0% (   0/4418 events) complete.
2023-03-01 21:38:14 13 [Note] WSREP: cluster conflict due to high priority abort for threads:
2023-03-01 21:38:14 13 [Note] WSREP: Winning thread:
   THD: 13, mode: applier, state: executing, conflict: no conflict, seqno: 5218150078
   SQL: DELETE FROM blobs WHERE id = '79562280'Ec
2023-03-01 21:38:14 13 [Note] WSREP: Victim thread:
   THD: 14, mode: applier, state: executing, conflict: no conflict, seqno: 5218150080
   SQL: DELETE FROM blobs WHERE id = '79562289'Ec
2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: BF Aborter BF thread: 13 seqno: 5218150078 query_state: executing conflict_state: no conflict exec mode applier query: DELETE FROM blobs WHERE id = '79562280'Ec
2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: Victim BF thread: 14 seqno: 5218150080 query_state: executing  conflict_state: must abort exec mode applier query: DELETE FROM blobs WHERE id = '79562289'Ec
2023-03-01 21:38:14 13 [ERROR] WSREP: Trx 5218150078 tries to abort slave trx 5218150080. This could be caused by:
        1) unsupported configuration options combination, please check documentation.
        2) a bug in the code.
        3) a database corruption.
 Node consistency compromized, need to abort. Restart the node to resync with cluster.
230301 21:38:14 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision: c73985f2ce8a391582787f3e310a011c1a712bec
key_buffer_size=1073741824
read_buffer_size=1073741824
max_used_connections=0
max_threads=1002
thread_count=15
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2102417504 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f6430000c08
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f71cc289e08 thread_stack 0x30000
2023-03-01 21:38:14 18 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
2023-03-01 21:38:14 19 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55745404c8fe]
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x557453ade925]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f75e7751980]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f75e738ce87]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f75e738e7f1]
/usr/sbin/mysqld(+0x4bc2ab)[0x5574538152ab]
/usr/sbin/mysqld(+0x903876)[0x557453c5c876]
/usr/sbin/mysqld(+0x954055)[0x557453cad055]
/usr/sbin/mysqld(+0x9590d1)[0x557453cb20d1]
/usr/sbin/mysqld(+0x95a3ae)[0x557453cb33ae]
/usr/sbin/mysqld(+0x9c4e0b)[0x557453d1de0b]
/usr/sbin/mysqld(+0xa01c28)[0x557453d5ac28]
/usr/sbin/mysqld(+0xa06d4b)[0x557453d5fd4b]
/usr/sbin/mysqld(+0xa090ac)[0x557453d620ac]
/usr/sbin/mysqld(+0x9da224)[0x557453d33224]
/usr/sbin/mysqld(+0x9134d8)[0x557453c6c4d8]
/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x557453ae990f]
/usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x557453bd148d]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x22f)[0x557453bc494f]
/usr/sbin/mysqld(wsrep_apply_cb+0x512)[0x557453a5b452]
src/trx_handle.cpp:312(galera::TrxHandle::apply(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_trx_meta const&) const)[0x7f75d59917ee]
src/replicator_smm.cpp:92(apply_trx_ws(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_cb_status (*)(void*, unsigned int, wsrep_trx_meta const*, bool*, bool), galera::TrxHandle const&, wsrep_trx_meta const&))[0x7f75d59a01a4]
src/replicator_smm.cpp:459(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandle*))[0x7f75d59a35d8]
src/gu_mutex.hpp:38(galera::ReplicatorSMM::recv_IST(void*))[0x7f75d59b6d3d]
src/replicator_smm.cpp:369(galera::ReplicatorSMM::async_recv(void*))[0x7f75d59a6b63]
src/wsrep_provider.cpp:271(galera_recv)[0x7f75d597b8eb]
/usr/sbin/mysqld(+0x7032a5)[0x557453a5c2a5]
/usr/sbin/mysqld(start_wsrep_THD+0x3ce)[0x557453a4d32e]
2023-03-01 21:38:15 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') turning message relay requesting off
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f75e77466db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f75e746f61f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f71b4033c03): DELETE FROM blobs WHERE id = '79562280'
 
Connection ID (thread ID): 13
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /srv/galera/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             257338               257338               processes
Max open files            102400               102400               files
Max locked memory         67108864             67108864             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       257338               257338               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
 
Kernel version: Linux version 4.15.0-204-generic (buildd@lcy02-amd64-030) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #215-Ubuntu SMP Fri Jan 20 18:24:59 UTC 2023
 
2023-03-01 21:38:26 0 [Note] Starting MariaDB 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision c73985f2ce8a391582787f3e310a011c1a712bec as process 62552
2023-03-01 21:38:26 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): Galera 25.3.37(rd0a7bd74) by Codership Oy <info@codership.com> loaded successfully.
2023-03-01 21:38:26 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2023-03-01 21:38:26 0 [Note] WSREP: Found saved state: 64b22b04-a147-11e8-a2be-1f2375c28574:-1, safe_to_bootstrap: 0
2023-03-01 21:38:26 0 [Note] WSREP: Passing config to GCS: base_dir = /srv/galera/mysql/; base_host = 192.168.201.201; base_port = 4567; cert.log_conflicts = YES; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/galera/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/galera/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 16G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 500; gcs.fc_master_slave = YES; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_t
2023-03-01 21:38:26 0 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
2023-03-01 21:38:26 0 [Note] WSREP: Assign initial position for certification: 5218150076, protocol version: -1
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_sst_grab()
2023-03-01 21:38:26 0 [Note] WSREP: Start replication
2023-03-01 21:38:26 0 [Note] WSREP: Setting initial position to 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
2023-03-01 21:38:26 0 [Note] WSREP: protonet asio version 0
2023-03-01 21:38:26 0 [Note] WSREP: Using CRC-32C for message checksums.
2023-03-01 21:38:26 0 [Note] WSREP: initializing ssl context
2023-03-01 21:38:26 0 [Note] WSREP: backend: asio
2023-03-01 21:38:26 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2023-03-01 21:38:26 0 [Note] WSREP: restore pc from disk successfully
2023-03-01 21:38:26 0 [Note] WSREP: GMCast version 0
2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
2023-03-01 21:38:26 0 [Note] WSREP: EVS version 0



 Comments   
Comment by Daniel Black [ 2023-03-01 ]

FYI I think a significant number of these fixes where in MDEV-29684 (10.4+). There's 1 more 10.3 release left before end of maintenance. It might be prudent to plan an upgrade soon if this is a risk item for you.

note https://mariadb.com/kb/en/upgrading-from-mariadb-103-to-mariadb-104-with-galera-cluster/ applies for over 10.4 too and than more than one major version can occur in a jump.

All 10.4+ use the same galera-4 version.

Comment by Daniel Black [ 2023-05-23 ]

Now that 10.3 has had its last release, are you encountering any problem with a 10.4+ MariaDB version?

Generated at Thu Feb 08 10:18:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.