2022-06-20 15:33:57 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ 220620 15:33:57 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=101 max_threads=1002 thread_count=82 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x49000 /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x560f7de0aa4e] ??:0(my_print_stacktrace)[0x560f7d817307] /lib64/libpthread.so.0(+0xf630)[0x7fb4976de630] /lib64/libc.so.6(gsignal+0x37)[0x7fb496b29387] :0(__GI_raise)[0x7fb496b2aa78] /usr/sbin/mariadbd(+0xe48490)[0x560f7dc86490] ??:0(std::pair >, bool> std::_Rb_tree, std::_Select1st >, std::less, std::allocator > >::_M_emplace_unique(unsigned long&, bool&&))[0x560f7dc40059] ??:0(void std::__introsort_loop(unsigned char**, unsigned char**, long))[0x560f7dd94245] ??:0(tpool::task::execute())[0x560f7dd952fb] ??:0(tpool::thread_pool_generic::worker_main(tpool::worker_data*))[0x560f7dd939a1] /lib64/libstdc++.so.6(+0xb5330)[0x7fb497278330] /lib64/libpthread.so.0(+0x7ea5)[0x7fb4976d6ea5] /lib64/libc.so.6(clone+0x6d)[0x7fb496bf1b0d] The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:16 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:16 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:16 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:16 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:16 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:16 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302933944394 2022-06-20 15:34:16 0 [Note] InnoDB: Starting final batch to recover 4036 pages from redo log. 2022-06-20 15:34:17 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52284516 2022-06-20 15:34:17 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:17 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:17 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:17 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:17 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:17 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933945386; transaction id 451834724 2022-06-20 15:34:17 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:17 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:17 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:17 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:17 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:17 0 [Note] Starting table crash recovery... 2022-06-20 15:34:17 0 [Note] Crash table recovery finished. 220620 15:34:17 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x558ecca75298 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fff04508430 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x558ecac9aa4e] ??:0(handle_fatal_signal)[0x558eca6a7307] sigaction.c:0(__restore_rt)[0x7f64ffef3630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7f64f48894e2] ??:0(ha_mcs::delete_table(char const*))[0x7f64f4884421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7f64f4884f2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x558eca6abc7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x558eca6b239b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x558eca6b26c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x558eca48f547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x558eca6b0bd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x558eca5349f9] ??:0(ddl_log_execute_recovery())[0x558eca535ca7] ??:0(unireg_abort)[0x558eca3b150b] ??:0(mysqld_main(int, char**))[0x558eca3b7ceb] ??:0(__libc_start_main)[0x7f64ff32a555] ??:0(_start)[0x558eca3abeb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x558ecb701800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:23 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:23 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:23 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:23 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:23 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:23 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:24 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:24 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:24 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:24 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:24 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:24 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933951956; transaction id 451834730 2022-06-20 15:34:24 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:24 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:24 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:24 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:24 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:24 0 [Note] Starting table crash recovery... 2022-06-20 15:34:24 0 [Note] Crash table recovery finished. 220620 15:34:24 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x55c42a957cd8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffcbe3a3ea0 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x55c426ea4a4e] ??:0(handle_fatal_signal)[0x55c4268b1307] sigaction.c:0(__restore_rt)[0x7fb23bab4630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7fb23044a4e2] ??:0(ha_mcs::delete_table(char const*))[0x7fb230445421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7fb230445f2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x55c4268b5c7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x55c4268bc39b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x55c4268bc6c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x55c426699547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x55c4268babd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x55c42673e9f9] ??:0(ddl_log_execute_recovery())[0x55c42673fca7] ??:0(unireg_abort)[0x55c4265bb50b] ??:0(mysqld_main(int, char**))[0x55c4265c1ceb] ??:0(__libc_start_main)[0x7fb23aeeb555] ??:0(_start)[0x55c4265b5eb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x55c42790b800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:30 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:30 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:30 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:30 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:30 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:30 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:30 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:31 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:31 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:31 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:31 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:31 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933953204; transaction id 451834730 2022-06-20 15:34:31 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:31 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:31 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:31 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:31 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:31 0 [Note] Starting table crash recovery... 2022-06-20 15:34:31 0 [Note] Crash table recovery finished. 220620 15:34:31 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x56419ce182a8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fff3f247410 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x56419a007a4e] ??:0(handle_fatal_signal)[0x564199a14307] sigaction.c:0(__restore_rt)[0x7f19e9a94630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7f19dccd04e2] ??:0(ha_mcs::delete_table(char const*))[0x7f19dcccb421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7f19dcccbf2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x564199a18c7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x564199a1f39b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x564199a1f6c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x5641997fc547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x564199a1dbd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x5641998a19f9] ??:0(ddl_log_execute_recovery())[0x5641998a2ca7] ??:0(unireg_abort)[0x56419971e50b] ??:0(mysqld_main(int, char**))[0x564199724ceb] ??:0(__libc_start_main)[0x7f19e8ecb555] ??:0(_start)[0x564199718eb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x56419aa6e800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:37 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:37 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:37 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:37 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:37 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:38 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:38 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:38 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:38 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:38 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:38 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:38 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:38 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933954436; transaction id 451834730 2022-06-20 15:34:38 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:38 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:38 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:38 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:38 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:38 0 [Note] Starting table crash recovery... 2022-06-20 15:34:38 0 [Note] Crash table recovery finished. 2022-06-20 15:34:38 0 [ERROR] DDL_LOG: Aborting executing entry 1 after 4 retries 220620 15:34:38 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x5633808f4e58 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffcb36c8e60 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x56337d694a4e] ??:0(handle_fatal_signal)[0x56337d0a1307] sigaction.c:0(__restore_rt)[0x7f3f8ccba630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7f3f754574e2] ??:0(ha_mcs::delete_table(char const*))[0x7f3f75452421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7f3f75452f2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x56337d0a5c7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x56337d0ac39b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x56337d0ac6c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x56337ce89547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x56337d0aabd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x56337cf2e9f9] ??:0(ddl_log_execute_recovery())[0x56337cf2fca7] ??:0(unireg_abort)[0x56337cdab50b] ??:0(mysqld_main(int, char**))[0x56337cdb1ceb] ??:0(__libc_start_main)[0x7f3f8c0f1555] ??:0(_start)[0x56337cda5eb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x56337e0fb800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:44 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:44 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:44 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:44 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:44 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:45 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:45 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:45 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:45 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:45 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:45 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933955684; transaction id 451834730 2022-06-20 15:34:45 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:45 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:45 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:45 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:45 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:45 0 [Note] Starting table crash recovery... 2022-06-20 15:34:45 0 [Note] Crash table recovery finished. 220620 15:34:45 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x55d3cc11edc8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fff498d2870 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x55d3c9f9ea4e] ??:0(handle_fatal_signal)[0x55d3c99ab307] sigaction.c:0(__restore_rt)[0x7f0fa5170630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7f0f983ac4e2] ??:0(ha_mcs::delete_table(char const*))[0x7f0f983a7421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7f0f983a7f2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x55d3c99afc7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x55d3c99b639b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x55d3c99b66c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x55d3c9793547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x55d3c99b4bd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x55d3c98389f9] ??:0(ddl_log_execute_recovery())[0x55d3c9839ca7] ??:0(unireg_abort)[0x55d3c96b550b] ??:0(mysqld_main(int, char**))[0x55d3c96bbceb] ??:0(__libc_start_main)[0x7f0fa45a7555] ??:0(_start)[0x55d3c96afeb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x55d3caa05800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:51 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:51 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:51 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:51 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:51 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:51 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:52 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:52 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:52 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:52 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:52 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933956916; transaction id 451834730 2022-06-20 15:34:52 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:52 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:52 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:52 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:52 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:52 0 [Note] Starting table crash recovery... 2022-06-20 15:34:52 0 [Note] Crash table recovery finished. 220620 15:34:52 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.6.7-3-MariaDB-enterprise-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2337567 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x5637419e4e58 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7ffc836990b0 thread_stack 0x49000 ??:0(my_print_stacktrace)[0x56373df56a4e] ??:0(handle_fatal_signal)[0x56373d963307] sigaction.c:0(__restore_rt)[0x7fe1c52cd630] ??:0(ha_mcs_impl_delete_table(char const*))[0x7fe1b85094e2] ??:0(ha_mcs::delete_table(char const*))[0x7fe1b8504421] ??:0(ha_mcs_cache::delete_table(char const*))[0x7fe1b8504f2e] ??:0(get_canonical_filename(handler*, char const*, char*))[0x56373d967c7f] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x56373d96e39b] ??:0(ha_delete_table(THD*, handlerton*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, bool))[0x56373d96e6c0] ??:0(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int*, void*), int, unsigned int, void*))[0x56373d74b547] ??:0(ha_delete_table_force(THD*, char const*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*))[0x56373d96cbd8] ??:0(JOIN::JOIN(THD*, List&, unsigned long long, select_result*))[0x56373d7f09f9] ??:0(ddl_log_execute_recovery())[0x56373d7f1ca7] ??:0(unireg_abort)[0x56373d66d50b] ??:0(mysqld_main(int, char**))[0x56373d673ceb] ??:0(__libc_start_main)[0x7fe1c4704555] ??:0(_start)[0x56373d667eb3] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x56373e9bd800): INTERNAL DDL LOG RECOVER IN PROGRESS Connection ID (thread ID): 0 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 482740 482740 processes Max open files 32768 32768 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 482740 482740 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core 220620 15:34:58 Columnstore: Started; Version: 6.2.3-1 2022-06-20 15:34:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2022-06-20 15:34:58 0 [Note] InnoDB: Number of pools: 1 2022-06-20 15:34:58 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2022-06-20 15:34:58 0 [Note] InnoDB: Using Linux native AIO 2022-06-20 15:34:58 0 [Note] InnoDB: Initializing buffer pool, total size = 19327352832, chunk size = 134217728 2022-06-20 15:34:58 0 [Note] InnoDB: Completed initialization of buffer pool 2022-06-20 15:34:58 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=2302925319028,2302925319028 2022-06-20 15:34:58 0 [Note] InnoDB: Starting final batch to recover 4038 pages from redo log. 2022-06-20 15:34:58 0 [Note] InnoDB: Last binlog file './sw-mcs-bin.000018', position 52808056 2022-06-20 15:34:58 0 [Note] InnoDB: 128 rollback segments are active. 2022-06-20 15:34:58 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" 2022-06-20 15:34:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-06-20 15:34:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-06-20 15:34:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-06-20 15:34:58 0 [Note] InnoDB: 10.6.7 started; log sequence number 2302933958164; transaction id 451834730 2022-06-20 15:34:58 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool 2022-06-20 15:34:58 0 [Note] Plugin 'FEEDBACK' is disabled. 2022-06-20 15:34:58 server_audit: MariaDB Audit Plugin version 2.4.1 STARTED. 2022-06-20 15:34:58 server_audit: Query cache is enabled with the TABLE events. Some table reads can be veiled. 2022-06-20 15:34:58 0 [Note] Recovering after a crash using sw-mcs-bin 2022-06-20 15:34:58 0 [Note] Starting table crash recovery... 2022-06-20 15:34:58 0 [Note] Crash table recovery finished. 2022-06-20 15:34:58 0 [ERROR] DDL_LOG: Aborting executing entry 29 after 4 retries 2022-06-20 15:34:58 0 [Note] Server socket created on IP: '0.0.0.0'. 2022-06-20 15:34:58 0 [Note] Server socket created on IP: '::'. 2022-06-20 15:34:59 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=sw-mcs2-relay-bin' to avoid this problem. 2022-06-20 15:34:59 5 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@10.10.180.25:3306' in log 'sw-mariadb10.015218' at position 4 2022-06-20 15:34:59 6 [Note] Slave SQL thread initialized, starting replication in log 'sw-mariadb10.015218' at position 4, relay log './sw-mcs2-relay-bin.000001' position: 4; GTID position '0-5-4195409243' 2022-06-20 15:34:59 0 [Note] /usr/sbin/mariadbd: ready for connections. Version: '10.6.7-3-MariaDB-enterprise-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Enterprise Server 2022-06-20 15:34:59 5 [Note] Slave I/O thread: connected to master 'repl@10.10.180.25:3306',replication starts at GTID position '0-5-4195409243'