MariaDB 10.3.38 Galera cluster nodes are crushing

2023-03-01 21:38:12 0 [Note] WSREP: Member 0.0 (mariadb1-u) requested state transfer from '*any*'. Selected 1.0 (mariadb4-u)(SYNCED) as donor.

2023-03-01 21:38:12 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5218154551)

2023-03-01 21:38:12 2 [Note] WSREP: Requesting state transfer: success, donor: 1

2023-03-01 21:38:12 2 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218154494

2023-03-01 21:38:12 0 [Note] WSREP: 1.0 (mariadb4-u): State transfer to 0.0 (mariadb1-u) complete.

2023-03-01 21:38:12 0 [Note] WSREP: Member 1.0 (mariadb4-u) synced with group.

2023-03-01 21:38:12 0 [Note] WSREP: SST complete, seqno: 5218150076

2023-03-01 21:38:12 0 [Note] Loaded 'ha_federated.so' with offset 0x7f71cd5b0000

2023-03-01 21:38:12 0 [Warning] Plugin 'FEDERATED' is of maturity level gamma while the server is stable

2023-03-01 21:38:12 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/

2023-03-01 21:38:12 0 [Note] InnoDB: Using Linux native AIO

2023-03-01 21:38:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2023-03-01 21:38:12 0 [Note] InnoDB: Uses event mutexes

2023-03-01 21:38:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2023-03-01 21:38:12 0 [Note] InnoDB: Number of pools: 1

2023-03-01 21:38:12 0 [Note] InnoDB: Using SSE2 crc32 instructions

2023-03-01 21:38:12 0 [Note] InnoDB: Initializing buffer pool, total size = 48G, instances = 8, chunk size = 128M

2023-03-01 21:38:13 0 [Note] InnoDB: Completed initialization of buffer pool

2023-03-01 21:38:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2023-03-01 21:38:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

2023-03-01 21:38:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2023-03-01 21:38:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2023-03-01 21:38:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

2023-03-01 21:38:13 0 [Note] InnoDB: 10.3.38 started; log sequence number 18289586082679; transaction id 17367262174

2023-03-01 21:38:13 0 [Note] InnoDB: Loading buffer pool(s) from /srv/galera/mysql/ib_buffer_pool

2023-03-01 21:38:13 0 [Note] Plugin 'FEEDBACK' is disabled.

2023-03-01 21:38:13 0 [Note] Server socket created on IP: '0.0.0.0'.

2023-03-01 21:38:13 0 [Note] WSREP: Signalling provider to continue.

2023-03-01 21:38:13 0 [Note] WSREP: SST received: 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076

2023-03-01 21:38:13 0 [Note] Reading of all Master_info entries succeeded

2023-03-01 21:38:13 0 [Note] Added new Master_info '' to hash table

2023-03-01 21:38:13 0 [Note] /usr/sbin/mysqld: ready for connections.

Version: '10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

2023-03-01 21:38:13 2 [Note] WSREP: Receiving IST: 4418 writesets, seqnos 5218150076-5218154494

2023-03-01 21:38:13 0 [Note] WSREP: Receiving IST...  0.0% (   0/4418 events) complete.

2023-03-01 21:38:14 13 [Note] WSREP: cluster conflict due to high priority abort for threads:

2023-03-01 21:38:14 13 [Note] WSREP: Winning thread:

   THD: 13, mode: applier, state: executing, conflict: no conflict, seqno: 5218150078

   SQL: DELETE FROM blobs WHERE id = '79562280'Ec

2023-03-01 21:38:14 13 [Note] WSREP: Victim thread:

   THD: 14, mode: applier, state: executing, conflict: no conflict, seqno: 5218150080

   SQL: DELETE FROM blobs WHERE id = '79562289'Ec

2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: BF Aborter BF thread: 13 seqno: 5218150078 query_state: executing conflict_state: no conflict exec mode applier query: DELETE FROM blobs WHERE id = '79562280'Ec

2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: Victim BF thread: 14 seqno: 5218150080 query_state: executing  conflict_state: must abort exec mode applier query: DELETE FROM blobs WHERE id = '79562289'Ec

2023-03-01 21:38:14 13 [ERROR] WSREP: Trx 5218150078 tries to abort slave trx 5218150080. This could be caused by:

        1) unsupported configuration options combination, please check documentation.

        2) a bug in the code.

        3) a database corruption.

 Node consistency compromized, need to abort. Restart the node to resync with cluster.

230301 21:38:14 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision: c73985f2ce8a391582787f3e310a011c1a712bec

key_buffer_size=1073741824

read_buffer_size=1073741824

max_used_connections=0

max_threads=1002

thread_count=15

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2102417504 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6430000c08

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x7f71cc289e08 thread_stack 0x30000

2023-03-01 21:38:14 18 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

2023-03-01 21:38:14 19 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55745404c8fe]

/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x557453ade925]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f75e7751980]

/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f75e738ce87]

/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f75e738e7f1]

/usr/sbin/mysqld(+0x4bc2ab)[0x5574538152ab]

/usr/sbin/mysqld(+0x903876)[0x557453c5c876]

/usr/sbin/mysqld(+0x954055)[0x557453cad055]

/usr/sbin/mysqld(+0x9590d1)[0x557453cb20d1]

/usr/sbin/mysqld(+0x95a3ae)[0x557453cb33ae]

/usr/sbin/mysqld(+0x9c4e0b)[0x557453d1de0b]

/usr/sbin/mysqld(+0xa01c28)[0x557453d5ac28]

/usr/sbin/mysqld(+0xa06d4b)[0x557453d5fd4b]

/usr/sbin/mysqld(+0xa090ac)[0x557453d620ac]

/usr/sbin/mysqld(+0x9da224)[0x557453d33224]

/usr/sbin/mysqld(+0x9134d8)[0x557453c6c4d8]

/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x557453ae990f]

/usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x557453bd148d]

/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x22f)[0x557453bc494f]

/usr/sbin/mysqld(wsrep_apply_cb+0x512)[0x557453a5b452]

src/trx_handle.cpp:312(galera::TrxHandle::apply(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_trx_meta const&) const)[0x7f75d59917ee]

src/replicator_smm.cpp:92(apply_trx_ws(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_cb_status (*)(void*, unsigned int, wsrep_trx_meta const*, bool*, bool), galera::TrxHandle const&, wsrep_trx_meta const&))[0x7f75d59a01a4]

src/replicator_smm.cpp:459(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandle*))[0x7f75d59a35d8]

src/gu_mutex.hpp:38(galera::ReplicatorSMM::recv_IST(void*))[0x7f75d59b6d3d]

src/replicator_smm.cpp:369(galera::ReplicatorSMM::async_recv(void*))[0x7f75d59a6b63]

src/wsrep_provider.cpp:271(galera_recv)[0x7f75d597b8eb]

/usr/sbin/mysqld(+0x7032a5)[0x557453a5c2a5]

/usr/sbin/mysqld(start_wsrep_THD+0x3ce)[0x557453a4d32e]

2023-03-01 21:38:15 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') turning message relay requesting off

/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f75e77466db]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f75e746f61f]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x7f71b4033c03): DELETE FROM blobs WHERE id = '79562280'

Connection ID (thread ID): 13

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /srv/galera/mysql

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             257338               257338               processes

Max open files            102400               102400               files

Max locked memory         67108864             67108864             bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       257338               257338               signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

Kernel version: Linux version 4.15.0-204-generic (buildd@lcy02-amd64-030) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #215-Ubuntu SMP Fri Jan 20 18:24:59 UTC 2023

2023-03-01 21:38:26 0 [Note] Starting MariaDB 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision c73985f2ce8a391582787f3e310a011c1a712bec as process 62552

2023-03-01 21:38:26 0 [Note] WSREP: Read nil XID from storage engines, skipping position init

2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'

2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): Galera 25.3.37(rd0a7bd74) by Codership Oy <info@codership.com> loaded successfully.

2023-03-01 21:38:26 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.

2023-03-01 21:38:26 0 [Note] WSREP: Found saved state: 64b22b04-a147-11e8-a2be-1f2375c28574:-1, safe_to_bootstrap: 0

2023-03-01 21:38:26 0 [Note] WSREP: Passing config to GCS: base_dir = /srv/galera/mysql/; base_host = 192.168.201.201; base_port = 4567; cert.log_conflicts = YES; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/galera/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/galera/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 16G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 500; gcs.fc_master_slave = YES; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_t

2023-03-01 21:38:26 0 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076

2023-03-01 21:38:26 0 [Note] WSREP: Assign initial position for certification: 5218150076, protocol version: -1

2023-03-01 21:38:26 0 [Note] WSREP: wsrep_sst_grab()

2023-03-01 21:38:26 0 [Note] WSREP: Start replication

2023-03-01 21:38:26 0 [Note] WSREP: Setting initial position to 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076

2023-03-01 21:38:26 0 [Note] WSREP: protonet asio version 0

2023-03-01 21:38:26 0 [Note] WSREP: Using CRC-32C for message checksums.

2023-03-01 21:38:26 0 [Note] WSREP: initializing ssl context

2023-03-01 21:38:26 0 [Note] WSREP: backend: asio

2023-03-01 21:38:26 0 [Note] WSREP: gcomm thread scheduling priority set to other:0

2023-03-01 21:38:26 0 [Note] WSREP: restore pc from disk successfully

2023-03-01 21:38:26 0 [Note] WSREP: GMCast version 0

2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567

2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') multicast: , ttl: 1

2023-03-01 21:38:26 0 [Note] WSREP: EVS version 0