2023-03-01 21:38:12 0 [Note] WSREP: Member 0.0 (mariadb1-u) requested state transfer from '*any*'. Selected 1.0 (mariadb4-u)(SYNCED) as donor.
|
2023-03-01 21:38:12 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5218154551)
|
2023-03-01 21:38:12 2 [Note] WSREP: Requesting state transfer: success, donor: 1
|
2023-03-01 21:38:12 2 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218154494
|
2023-03-01 21:38:12 0 [Note] WSREP: 1.0 (mariadb4-u): State transfer to 0.0 (mariadb1-u) complete.
|
2023-03-01 21:38:12 0 [Note] WSREP: Member 1.0 (mariadb4-u) synced with group.
|
2023-03-01 21:38:12 0 [Note] WSREP: SST complete, seqno: 5218150076
|
2023-03-01 21:38:12 0 [Note] Loaded 'ha_federated.so' with offset 0x7f71cd5b0000
|
2023-03-01 21:38:12 0 [Warning] Plugin 'FEDERATED' is of maturity level gamma while the server is stable
|
2023-03-01 21:38:12 0 [Warning] The parameter innodb_file_format is deprecated and has no effect. It may be removed in future releases. See https://mariadb.com/kb/en/library/xtradbinnodb-file-format/
|
2023-03-01 21:38:12 0 [Note] InnoDB: Using Linux native AIO
|
2023-03-01 21:38:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2023-03-01 21:38:12 0 [Note] InnoDB: Uses event mutexes
|
2023-03-01 21:38:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2023-03-01 21:38:12 0 [Note] InnoDB: Number of pools: 1
|
2023-03-01 21:38:12 0 [Note] InnoDB: Using SSE2 crc32 instructions
|
2023-03-01 21:38:12 0 [Note] InnoDB: Initializing buffer pool, total size = 48G, instances = 8, chunk size = 128M
|
2023-03-01 21:38:13 0 [Note] InnoDB: Completed initialization of buffer pool
|
2023-03-01 21:38:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2023-03-01 21:38:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
2023-03-01 21:38:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2023-03-01 21:38:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2023-03-01 21:38:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2023-03-01 21:38:13 0 [Note] InnoDB: 10.3.38 started; log sequence number 18289586082679; transaction id 17367262174
|
2023-03-01 21:38:13 0 [Note] InnoDB: Loading buffer pool(s) from /srv/galera/mysql/ib_buffer_pool
|
2023-03-01 21:38:13 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2023-03-01 21:38:13 0 [Note] Server socket created on IP: '0.0.0.0'.
|
2023-03-01 21:38:13 0 [Note] WSREP: Signalling provider to continue.
|
2023-03-01 21:38:13 0 [Note] WSREP: SST received: 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
|
2023-03-01 21:38:13 0 [Note] Reading of all Master_info entries succeeded
|
2023-03-01 21:38:13 0 [Note] Added new Master_info '' to hash table
|
2023-03-01 21:38:13 0 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
|
2023-03-01 21:38:13 2 [Note] WSREP: Receiving IST: 4418 writesets, seqnos 5218150076-5218154494
|
2023-03-01 21:38:13 0 [Note] WSREP: Receiving IST... 0.0% ( 0/4418 events) complete.
|
2023-03-01 21:38:14 13 [Note] WSREP: cluster conflict due to high priority abort for threads:
|
2023-03-01 21:38:14 13 [Note] WSREP: Winning thread:
|
THD: 13, mode: applier, state: executing, conflict: no conflict, seqno: 5218150078
|
SQL: DELETE FROM blobs WHERE id = '79562280'Ec
|
2023-03-01 21:38:14 13 [Note] WSREP: Victim thread:
|
THD: 14, mode: applier, state: executing, conflict: no conflict, seqno: 5218150080
|
SQL: DELETE FROM blobs WHERE id = '79562289'Ec
|
2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: BF Aborter BF thread: 13 seqno: 5218150078 query_state: executing conflict_state: no conflict exec mode applier query: DELETE FROM blobs WHERE id = '79562280'Ec
|
2023-03-01 21:38:14 13 [ERROR] WSREP: wsrep_abort_slave_trx: Victim BF thread: 14 seqno: 5218150080 query_state: executing conflict_state: must abort exec mode applier query: DELETE FROM blobs WHERE id = '79562289'Ec
|
2023-03-01 21:38:14 13 [ERROR] WSREP: Trx 5218150078 tries to abort slave trx 5218150080. This could be caused by:
|
1) unsupported configuration options combination, please check documentation.
|
2) a bug in the code.
|
3) a database corruption.
|
Node consistency compromized, need to abort. Restart the node to resync with cluster.
|
230301 21:38:14 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision: c73985f2ce8a391582787f3e310a011c1a712bec
|
key_buffer_size=1073741824
|
read_buffer_size=1073741824
|
max_used_connections=0
|
max_threads=1002
|
thread_count=15
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2102417504 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f6430000c08
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f71cc289e08 thread_stack 0x30000
|
2023-03-01 21:38:14 18 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
|
2023-03-01 21:38:14 19 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55745404c8fe]
|
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x557453ade925]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f75e7751980]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f75e738ce87]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f75e738e7f1]
|
/usr/sbin/mysqld(+0x4bc2ab)[0x5574538152ab]
|
/usr/sbin/mysqld(+0x903876)[0x557453c5c876]
|
/usr/sbin/mysqld(+0x954055)[0x557453cad055]
|
/usr/sbin/mysqld(+0x9590d1)[0x557453cb20d1]
|
/usr/sbin/mysqld(+0x95a3ae)[0x557453cb33ae]
|
/usr/sbin/mysqld(+0x9c4e0b)[0x557453d1de0b]
|
/usr/sbin/mysqld(+0xa01c28)[0x557453d5ac28]
|
/usr/sbin/mysqld(+0xa06d4b)[0x557453d5fd4b]
|
/usr/sbin/mysqld(+0xa090ac)[0x557453d620ac]
|
/usr/sbin/mysqld(+0x9da224)[0x557453d33224]
|
/usr/sbin/mysqld(+0x9134d8)[0x557453c6c4d8]
|
/usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x557453ae990f]
|
/usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEP14rpl_group_info+0x12d)[0x557453bd148d]
|
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x22f)[0x557453bc494f]
|
/usr/sbin/mysqld(wsrep_apply_cb+0x512)[0x557453a5b452]
|
src/trx_handle.cpp:312(galera::TrxHandle::apply(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_trx_meta const&) const)[0x7f75d59917ee]
|
src/replicator_smm.cpp:92(apply_trx_ws(void*, wsrep_cb_status (*)(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*), wsrep_cb_status (*)(void*, unsigned int, wsrep_trx_meta const*, bool*, bool), galera::TrxHandle const&, wsrep_trx_meta const&))[0x7f75d59a01a4]
|
src/replicator_smm.cpp:459(galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandle*))[0x7f75d59a35d8]
|
src/gu_mutex.hpp:38(galera::ReplicatorSMM::recv_IST(void*))[0x7f75d59b6d3d]
|
src/replicator_smm.cpp:369(galera::ReplicatorSMM::async_recv(void*))[0x7f75d59a6b63]
|
src/wsrep_provider.cpp:271(galera_recv)[0x7f75d597b8eb]
|
/usr/sbin/mysqld(+0x7032a5)[0x557453a5c2a5]
|
/usr/sbin/mysqld(start_wsrep_THD+0x3ce)[0x557453a4d32e]
|
2023-03-01 21:38:15 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') turning message relay requesting off
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f75e77466db]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f75e746f61f]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f71b4033c03): DELETE FROM blobs WHERE id = '79562280'
|
|
Connection ID (thread ID): 13
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /srv/galera/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 257338 257338 processes
|
Max open files 102400 102400 files
|
Max locked memory 67108864 67108864 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 257338 257338 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
|
|
Kernel version: Linux version 4.15.0-204-generic (buildd@lcy02-amd64-030) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #215-Ubuntu SMP Fri Jan 20 18:24:59 UTC 2023
|
|
2023-03-01 21:38:26 0 [Note] Starting MariaDB 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision c73985f2ce8a391582787f3e310a011c1a712bec as process 62552
|
2023-03-01 21:38:26 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
|
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
|
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_load(): Galera 25.3.37(rd0a7bd74) by Codership Oy <info@codership.com> loaded successfully.
|
2023-03-01 21:38:26 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
|
2023-03-01 21:38:26 0 [Note] WSREP: Found saved state: 64b22b04-a147-11e8-a2be-1f2375c28574:-1, safe_to_bootstrap: 0
|
2023-03-01 21:38:26 0 [Note] WSREP: Passing config to GCS: base_dir = /srv/galera/mysql/; base_host = 192.168.201.201; base_port = 4567; cert.log_conflicts = YES; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /srv/galera/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /srv/galera/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 16G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 500; gcs.fc_master_slave = YES; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_t
|
2023-03-01 21:38:26 0 [Note] WSREP: GCache history reset: 64b22b04-a147-11e8-a2be-1f2375c28574:0 -> 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
|
2023-03-01 21:38:26 0 [Note] WSREP: Assign initial position for certification: 5218150076, protocol version: -1
|
2023-03-01 21:38:26 0 [Note] WSREP: wsrep_sst_grab()
|
2023-03-01 21:38:26 0 [Note] WSREP: Start replication
|
2023-03-01 21:38:26 0 [Note] WSREP: Setting initial position to 64b22b04-a147-11e8-a2be-1f2375c28574:5218150076
|
2023-03-01 21:38:26 0 [Note] WSREP: protonet asio version 0
|
2023-03-01 21:38:26 0 [Note] WSREP: Using CRC-32C for message checksums.
|
2023-03-01 21:38:26 0 [Note] WSREP: initializing ssl context
|
2023-03-01 21:38:26 0 [Note] WSREP: backend: asio
|
2023-03-01 21:38:26 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
|
2023-03-01 21:38:26 0 [Note] WSREP: restore pc from disk successfully
|
2023-03-01 21:38:26 0 [Note] WSREP: GMCast version 0
|
2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
|
2023-03-01 21:38:26 0 [Note] WSREP: (4d43ade6, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
|
2023-03-01 21:38:26 0 [Note] WSREP: EVS version 0
|