2021-05-20 13:21:39 0 [Note] WSREP: Loading provider /usr/lib64/galera-4/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1 2021-05-20 13:21:39 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-4/libgalera_smm.so' 2021-05-20 13:21:39 0 [Note] WSREP: wsrep_load(): Galera 26.4.8(r902dd26) by Codership Oy loaded successfully. 2021-05-20 13:21:39 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration. 2021-05-20 13:21:39 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat' 2021-05-20 13:21:39 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1 2021-05-20 13:21:39 0 [Note] WSREP: GCache DEBUG: opened preamble: Version: 0 UUID: 00000000-0000-0000-0000-000000000000 Seqno: -1 - -1 Offset: -1 Synced: 0 2021-05-20 13:21:39 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID. 2021-05-20 13:21:39 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.0.40; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 1G; gcache.recover = yes; gcache.size = 1G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; 2021-05-20 13:21:39 0 [Note] WSREP: Start replication 2021-05-20 13:21:39 0 [Note] WSREP: Connecting with bootstrap option: 0 2021-05-20 13:21:39 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 2021-05-20 13:21:39 0 [Note] WSREP: protonet asio version 0 2021-05-20 13:21:39 0 [Note] WSREP: Using CRC-32C for message checksums. 2021-05-20 13:21:39 0 [Note] WSREP: backend: asio 2021-05-20 13:21:39 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 2021-05-20 13:21:39 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) 2021-05-20 13:21:39 0 [Note] WSREP: restore pc from disk failed 2021-05-20 13:21:39 0 [Note] WSREP: GMCast version 0 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2021-05-20 13:21:39 0 [Note] WSREP: EVS version 1 2021-05-20 13:21:39 0 [Note] WSREP: gcomm: connecting to group 'services-db', peer '10.0.0.251:,10.0.0.24:,10.0.0.235:,10.0.0.40:' 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') Found matching local endpoint for a connection, blacklisting address tcp://10.0.0.40:4567 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') connection established to e638ef0a-8d74 tcp://10.0.0.24:4567 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') connection established to d8011642-93f3 tcp://10.0.0.251:4567 2021-05-20 13:21:39 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') connection established to e90dd5c4-a471 tcp://10.0.0.235:4567 2021-05-20 13:21:39 0 [Note] WSREP: EVS version upgrade 0 -> 1 2021-05-20 13:21:39 0 [Note] WSREP: declaring d8011642-93f3 at tcp://10.0.0.251:4567 stable 2021-05-20 13:21:39 0 [Note] WSREP: declaring e638ef0a-8d74 at tcp://10.0.0.24:4567 stable 2021-05-20 13:21:39 0 [Note] WSREP: declaring e90dd5c4-a471 at tcp://10.0.0.235:4567 stable 2021-05-20 13:21:39 0 [Note] WSREP: PC protocol upgrade 0 -> 1 2021-05-20 13:21:40 0 [Note] WSREP: Node d8011642-93f3 state prim 2021-05-20 13:21:40 0 [Note] WSREP: view(view_id(PRIM,4f576f70-9364,8) memb { 4f576f70-9364,0 d8011642-93f3,0 e638ef0a-8d74,0 e90dd5c4-a471,0 } joined { } left { } partitioned { }) 2021-05-20 13:21:40 0 [Note] WSREP: save pc into disk 2021-05-20 13:21:40 0 [Note] WSREP: gcomm: connected 2021-05-20 13:21:40 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 2021-05-20 13:21:40 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) 2021-05-20 13:21:40 0 [Note] WSREP: Opened channel 'services-db' 2021-05-20 13:21:40 1 [Note] WSREP: Starting rollbacker thread 1 2021-05-20 13:21:40 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 4 2021-05-20 13:21:40 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 503cc27a-b96e-11eb-948c-679d2134616e 2021-05-20 13:21:40 2 [Note] WSREP: Starting applier thread 2 2021-05-20 13:21:40 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 503cc27a-b96e-11eb-948c-679d2134616e 2021-05-20 13:21:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 503cc27a-b96e-11eb-948c-679d2134616e from 0 (services-db-a@20210520-1317) 2021-05-20 13:21:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 503cc27a-b96e-11eb-948c-679d2134616e from 1 (services-db-a@20210512-0844) 2021-05-20 13:21:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 503cc27a-b96e-11eb-948c-679d2134616e from 2 (services-db-a@20210512-0920) 2021-05-20 13:21:40 0 [Note] WSREP: STATE EXCHANGE: got state msg: 503cc27a-b96e-11eb-948c-679d2134616e from 3 (services-db-a@20210512-0942) 2021-05-20 13:21:40 0 [Note] WSREP: Quorum results: version = 6, component = PRIMARY, conf_id = 7, members = 3/4 (joined/total), act_id = 1381476, last_appl. = 1381355, protocols = 2/10/4 (gcs/repl/appl), vote policy= 0, group UUID = 5c748f27-b2f9-11eb-b78e-f73d161cbf58 2021-05-20 13:21:40 0 [Note] WSREP: Flow-control interval: [32, 32] 2021-05-20 13:21:40 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 1381477) 2021-05-20 13:21:40 2 [Note] WSREP: ####### processing CC 1381477, local, ordered 2021-05-20 13:21:40 2 [Note] WSREP: Process first view: 5c748f27-b2f9-11eb-b78e-f73d161cbf58 my uuid: 4f576f70-b96e-11eb-9364-52850b7a2064 2021-05-20 13:21:40 2 [Note] WSREP: Server services-db-a@20210520-1317 connected to cluster at position 5c748f27-b2f9-11eb-b78e-f73d161cbf58:1381477 with ID 4f576f70-b96e-11eb-9364-52850b7a2064 2021-05-20 13:21:40 2 [Note] WSREP: Server status change disconnected -> connected 2021-05-20 13:21:40 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-05-20 13:21:40 2 [Note] WSREP: ####### My UUID: 4f576f70-b96e-11eb-9364-52850b7a2064 2021-05-20 13:21:40 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes 2021-05-20 13:21:40 0 [Note] WSREP: Service thread queue flushed. 2021-05-20 13:21:40 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1 2021-05-20 13:21:40 2 [Note] WSREP: State transfer required: Group state: 5c748f27-b2f9-11eb-b78e-f73d161cbf58:1381477 Local state: 00000000-0000-0000-0000-000000000000:-1 2021-05-20 13:21:40 2 [Note] WSREP: Server status change connected -> joiner 2021-05-20 13:21:40 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-05-20 13:21:40 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.0.0.40' --datadir '/var/lib/mysql/' --parent '4057' --mysqld-args --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1' 2021-05-20 13:21:40 0 [Note] WSREP: Joiner monitor thread started to monitor 2021-05-20 13:21:40 2 [Note] WSREP: Prepared SST request: rsync|10.0.0.40:4444/rsync_sst 2021-05-20 13:21:40 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 1381477, STRv: 3 2021-05-20 13:21:40 2 [Note] WSREP: IST receiver addr using tcp://10.0.0.40:4568 2021-05-20 13:21:40 2 [Note] WSREP: Prepared IST receiver for 0-1381477, listening at: tcp://10.0.0.40:4568 2021-05-20 13:21:40 0 [Note] WSREP: Member 0.0 (services-db-a@20210520-1317) requested state transfer from '*any*'. Selected 1.0 (services-db-a@20210512-0844)(SYNCED) as donor. 2021-05-20 13:21:40 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1381479) 2021-05-20 13:21:40 2 [Note] WSREP: Requesting state transfer: success, donor: 1 2021-05-20 13:21:40 2 [Note] WSREP: Resetting GCache seqno map due to different histories. 2021-05-20 13:21:40 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 5c748f27-b2f9-11eb-b78e-f73d161cbf58:1381477 2021-05-20 13:21:42 0 [Note] WSREP: (4f576f70-9364, 'tcp://0.0.0.0:4567') turning message relay requesting off 2021-05-20 13:23:58 0 [Note] WSREP: 1.0 (services-db-a@20210512-0844): State transfer to 0.0 (services-db-a@20210520-1317) complete. 2021-05-20 13:23:58 0 [Note] WSREP: Member 1.0 (services-db-a@20210512-0844) synced with group. WSREP_SST: [INFO] Joiner cleanup. rsync PID: 4116 (20210520 13:23:59.416) WSREP_SST: [INFO] Joiner cleanup done. (20210520 13:23:59.923) 2021-05-20 13:23:59 3 [Note] WSREP: SST received 2021-05-20 13:23:59 3 [Note] WSREP: Server status change joiner -> initializing 2021-05-20 13:23:59 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2021-05-20 13:23:59 0 [Note] mariadbd: Aria engine: starting recovery recovered pages: 0% 10% 21% 31% 42% 53% 63% 73% 87% 97% 100% (0.0 seconds); tables to flush: 2 1 0 (0.0 seconds); 2021-05-20 13:23:59 0 [Note] mariadbd: Aria engine: recovery done 2021-05-20 13:23:59 0 [Note] InnoDB: Uses event mutexes 2021-05-20 13:23:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2021-05-20 13:23:59 0 [Note] InnoDB: Number of pools: 1 2021-05-20 13:23:59 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2021-05-20 13:23:59 0 [Note] InnoDB: Using Linux native AIO 2021-05-20 13:23:59 0 [Note] InnoDB: Initializing buffer pool, total size = 2684354560, chunk size = 134217728 2021-05-20 13:24:00 0 [Note] InnoDB: Completed initialization of buffer pool 2021-05-20 13:24:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=18950648175,18950795112 2021-05-20 13:24:02 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo 2021-05-20 13:24:02 0 [Note] InnoDB: Trx id counter is 2776768 2021-05-20 13:24:02 0 [Note] InnoDB: Starting final batch to recover 28591 pages from redo log. 2021-05-20 13:24:02 0 [ERROR] InnoDB: Not applying INSERT_REUSE_REDUNDANT due to corruption on [page id: space=0, page number=2366] 2021-05-20 13:24:02 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption. 2021-05-20 13:24:09 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption 2021-05-20 13:24:09 0 [Note] InnoDB: Starting shutdown... 2021-05-20 13:24:09 0 [ERROR] Plugin 'InnoDB' init function returned error. 2021-05-20 13:24:09 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2021-05-20 13:24:09 0 [Note] Plugin 'FEEDBACK' is disabled. 2021-05-20 13:24:09 0 [ERROR] Unknown/unsupported storage engine: InnoDB 2021-05-20 13:24:09 0 [ERROR] Aborting terminate called after throwing an instance of 'wsrep::runtime_error' what(): State wait was interrupted 210520 13:24:09 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.5.10-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=152 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 465669 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7efc74002078 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7efc7b5fda20 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x560bfd0e2dfe] ??:0(handle_fatal_signal)[0x560bfcae33b7] sigaction.c:0(__restore_rt)[0x7efce16017e0] :0(__GI_raise)[0x7efce09b2c20] :0(__GI_abort)[0x7efce09b40c8] ??:0(__gnu_cxx::__verbose_terminate_handler())[0x7efce10fc0ef] ??:0(std::rethrow_exception(std::__exception_ptr::exception_ptr))[0x7efce10f9e86] ??:0(std::terminate())[0x7efce10f9ed1] ??:0(__cxa_throw)[0x7efce10fa104] ??:0(wsrep::server_state::wait_until_state(std::unique_lock&, wsrep::server_state::state) const)[0x560bfd160384] ??:0(wsrep::server_state::sst_received(wsrep::client_service&, int))[0x560bfd1691dc] ??:0(void std::vector >::_M_range_insert<__gnu_cxx::__normal_iterator > > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, std::forward_iterato??:0(void std::vector >::_M_range_insert<__gnu_cxx::__normal_iterator > > >(__gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, std::forward_iterato??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x560bfcd34f3d] pthread_create.c:0(start_thread)[0x7efce15f740b] :0(__GI___clone)[0x7efce0a6c09f] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): (null) Connection ID (thread ID): 3 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. We think the query pointer is invalid, but we will try to print it anyway. Query: Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 10485760 bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 50000 50000 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 30446 30446 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core