2020-08-04 15:27:07 0 [Note] InnoDB: Using Linux native AIO 2020-08-04 15:27:07 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-04 15:27:07 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-04 15:27:07 0 [Note] InnoDB: Uses event mutexes 2020-08-04 15:27:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-04 15:27:07 0 [Note] InnoDB: Number of pools: 1 2020-08-04 15:27:07 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-04 15:27:07 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-04 15:27:10 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-04 15:27:10 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-04 15:27:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-04 15:27:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-04 15:27:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-04 15:27:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-04 15:27:13 0 [Note] InnoDB: Waiting for purge to start 2020-08-04 15:27:13 0 [Note] InnoDB: 10.3.23 started; log sequence number 678223185; transaction id 21353 2020-08-04 15:27:13 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-04 15:27:13 0 [Note] Plugin 'FEEDBACK' is disabled. 200804 15:27:13 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200804 15:27:13 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-04 15:27:13 0 [Note] Semi-sync replication initialized for transactions. 2020-08-04 15:27:13 0 [Note] Semi-sync replication enabled on the master. 2020-08-04 15:27:13 0 [Note] Starting ack receiver thread 2020-08-04 15:27:13 0 [Warning] Failed to setup SSL 2020-08-04 15:27:13 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-04 15:27:13 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-04 15:27:13 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-04 15:27:13 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-04 15:27:13 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-04 15:27:13 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-04 15:27:23 0 [Note] InnoDB: Buffer pool(s) load completed at 200804 15:27:23 2020-08-04 15:32:29 50 [Warning] Timeout waiting for reply of binlog (file: bin_log.000147, pos: 1006), semi-sync up to file , position 0. 2020-08-04 15:32:29 50 [Note] Semi-sync replication switched OFF. 2020-08-04 15:33:44 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-08-04 15:33:44 0 [Note] InnoDB: FTS optimize thread exiting. 2020-08-04 15:33:44 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-08-04 15:33:44 6 [Note] Stopping ack receiver thread 200804 15:33:46 server_audit: STOPPED 2020-08-04 15:33:48 0 [Note] InnoDB: Starting shutdown... 2020-08-04 15:33:48 0 [Note] InnoDB: Dumping buffer pool(s) to /mariadb/data/ib_buffer_pool 2020-08-04 15:33:48 0 [Note] InnoDB: Buffer pool(s) dump completed at 200804 15:33:48 2020-08-04 15:33:51 0 [Note] InnoDB: Shutdown completed; log sequence number 678241520; transaction id 21839 2020-08-04 15:33:51 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-04 15:33:52 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-08-04 15:34:23 0 [Note] InnoDB: Using Linux native AIO 2020-08-04 15:34:23 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-04 15:34:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-04 15:34:23 0 [Note] InnoDB: Uses event mutexes 2020-08-04 15:34:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-04 15:34:23 0 [Note] InnoDB: Number of pools: 1 2020-08-04 15:34:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-04 15:34:23 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-04 15:34:26 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-04 15:34:26 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-04 15:34:27 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-04 15:34:27 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-04 15:34:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-04 15:34:27 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-04 15:34:27 0 [Note] InnoDB: Waiting for purge to start 2020-08-04 15:34:27 0 [Note] InnoDB: 10.3.23 started; log sequence number 678241520; transaction id 21841 2020-08-04 15:34:27 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-04 15:34:27 0 [Note] Plugin 'FEEDBACK' is disabled. 200804 15:34:27 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200804 15:34:27 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-04 15:34:27 0 [Note] Semi-sync replication initialized for transactions. 2020-08-04 15:34:27 0 [Note] Semi-sync replication enabled on the master. 2020-08-04 15:34:27 0 [Note] Starting ack receiver thread 2020-08-04 15:34:33 0 [Warning] Failed to setup SSL 2020-08-04 15:34:33 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-04 15:34:33 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-04 15:34:33 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-04 15:34:33 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-04 15:34:33 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-04 15:34:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-04 15:34:34 0 [Note] InnoDB: Buffer pool(s) load completed at 200804 15:34:34 2020-08-04 15:45:21 107 [Warning] Timeout waiting for reply of binlog (file: bin_log.000148, pos: 660), semi-sync up to file , position 0. 2020-08-04 15:45:21 107 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200805 10:27:17 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=93 max_threads=1002 thread_count=100 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fbb4c0589b0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fbbb824bd58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x55fd4a086309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55fd4994967f] sigaction.c:0(__restore_rt)[0x7fc135d24630] :0(__GI_raise)[0x7fc133bd9377] :0(__GI_abort)[0x7fc133bdaa68] :0(__assert_fail_base)[0x7fc133bd2196] :0(__GI___assert_fail)[0x7fc133bd2242] /usr/sbin/mysqld(+0xeaef93)[0x55fd49f75f93] /usr/sbin/mysqld(+0xeb1b2c)[0x55fd49f78b2c] /usr/sbin/mysqld(+0xeb454f)[0x55fd49f7b54f] /usr/sbin/mysqld(+0xea904c)[0x55fd49f7004c] /usr/sbin/mysqld(+0xeaae74)[0x55fd49f71e74] /usr/sbin/mysqld(+0xeb7805)[0x55fd49f7e805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x55fd497764c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x55fd497802ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x55fd4978093f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x55fd497849e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x55fd49767ffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x55fd497688b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x55fd49766829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x55fd49767333] /usr/sbin/mysqld(+0x4f5135)[0x55fd495bc135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x55fd496fffb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x55fd49706cea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x55fd497089f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x55fd4970afef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x55fd4980d077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x55fd4980d134] pthread_create.c:0(start_thread)[0x7fc135d1cea5] /lib64/libc.so.6(clone+0x6d)[0x7fc133ca18cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fbb4c08f328): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 14903 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-05 10:28:46 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 10:28:46 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 10:28:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 10:28:46 0 [Note] InnoDB: Uses event mutexes 2020-08-05 10:28:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 10:28:46 0 [Note] InnoDB: Number of pools: 1 2020-08-05 10:28:46 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 10:28:46 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 10:28:49 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 10:28:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 10:28:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=679823390 2020-08-05 10:29:03 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000152', position 808412 2020-08-05 10:29:03 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 10:29:03 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 10:29:03 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 10:29:03 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 10:29:03 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 10:29:03 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 10:29:03 0 [Note] InnoDB: 10.3.23 started; log sequence number 679823399; transaction id 26691 2020-08-05 10:29:03 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 10:29:03 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 10:29:03 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 10:29:03 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 10:29:03 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 10:29:03 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 10:29:03 0 [Note] Starting ack receiver thread 2020-08-05 10:29:03 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-05 10:29:03 0 [Note] Starting crash recovery... 2020-08-05 10:29:04 0 [Note] Crash recovery finished. 2020-08-05 10:29:24 0 [Warning] Failed to setup SSL 2020-08-05 10:29:24 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 10:29:24 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 10:29:24 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 10:29:24 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 10:29:24 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 10:29:29 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 10:30:05 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 10:30:05 mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200805 10:39:31 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=69 max_threads=1002 thread_count=77 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f065c01c8a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0685736d58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x558576e95309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55857675867f] sigaction.c:0(__restore_rt)[0x7f0c0c7b6630] :0(__GI_raise)[0x7f0c0a66b377] :0(__GI_abort)[0x7f0c0a66ca68] :0(__assert_fail_base)[0x7f0c0a664196] :0(__GI___assert_fail)[0x7f0c0a664242] /usr/sbin/mysqld(+0xeaef93)[0x558576d84f93] /usr/sbin/mysqld(+0xeb1b2c)[0x558576d87b2c] /usr/sbin/mysqld(+0xeb454f)[0x558576d8a54f] /usr/sbin/mysqld(+0xea904c)[0x558576d7f04c] /usr/sbin/mysqld(+0xeaae74)[0x558576d80e74] /usr/sbin/mysqld(+0xeb7805)[0x558576d8d805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x5585765854c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x55857658f2ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x55857658f93f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x5585765939e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x558576576ffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x5585765778b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x558576575829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x558576576333] /usr/sbin/mysqld(+0x4f5135)[0x5585763cb135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x55857650efb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x558576515cea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x5585765179f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x558576519fef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x55857661c077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x55857661c134] pthread_create.c:0(start_thread)[0x7f0c0c7aeea5] /lib64/libc.so.6(clone+0x6d)[0x7f0c0a7338cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f065c02d648): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 158 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-05 10:40:56 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 10:40:56 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 10:40:56 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 10:40:56 0 [Note] InnoDB: Uses event mutexes 2020-08-05 10:40:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 10:40:56 0 [Note] InnoDB: Number of pools: 1 2020-08-05 10:40:56 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 10:40:56 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 10:41:00 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 10:41:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 10:41:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=679823408 2020-08-05 10:41:01 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000152', position 808412 2020-08-05 10:41:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 10:41:01 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 10:41:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 10:41:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 10:41:01 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 10:41:01 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 10:41:01 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 10:41:01 0 [Note] InnoDB: 10.3.23 started; log sequence number 679823417; transaction id 26691 2020-08-05 10:41:01 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 10:41:01 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 10:41:01 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 10:41:01 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 10:41:01 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 10:41:01 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 10:41:01 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-05 10:41:01 0 [Note] Starting crash recovery... 2020-08-05 10:41:01 0 [Note] Starting ack receiver thread 2020-08-05 10:41:01 0 [Note] Crash recovery finished. 2020-08-05 10:41:02 0 [Warning] Failed to setup SSL 2020-08-05 10:41:02 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 10:41:02 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 10:41:02 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 10:41:02 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 10:41:02 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 10:41:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 10:41:09 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 10:41:09 2020-08-05 10:56:13 50 [Warning] Timeout waiting for reply of binlog (file: bin_log.000154, pos: 1006), semi-sync up to file , position 0. 2020-08-05 10:56:13 50 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200805 11:42:00 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=92 max_threads=1002 thread_count=101 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f0de001c6e0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f0e703c3d58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x56321a3cd309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563219c9067f] sigaction.c:0(__restore_rt)[0x7f13e8ec0630] :0(__GI_raise)[0x7f13e6d75377] :0(__GI_abort)[0x7f13e6d76a68] :0(__assert_fail_base)[0x7f13e6d6e196] :0(__GI___assert_fail)[0x7f13e6d6e242] /usr/sbin/mysqld(+0xeaef93)[0x56321a2bcf93] /usr/sbin/mysqld(+0xeb1b2c)[0x56321a2bfb2c] /usr/sbin/mysqld(+0xeb454f)[0x56321a2c254f] /usr/sbin/mysqld(+0xea904c)[0x56321a2b704c] /usr/sbin/mysqld(+0xeaae74)[0x56321a2b8e74] /usr/sbin/mysqld(+0xeb7805)[0x56321a2c5805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x563219abd4c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x563219ac72ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x563219ac793f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x563219acb9e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x563219aaeffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x563219aaf8b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x563219aad829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x563219aae333] /usr/sbin/mysqld(+0x4f5135)[0x563219903135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x563219a46fb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x563219a4dcea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x563219a4f9f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x563219a51fef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x563219b54077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x563219b54134] pthread_create.c:0(start_thread)[0x7f13e8eb8ea5] /lib64/libc.so.6(clone+0x6d)[0x7f13e6e3d8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f0de002d5b8): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 700 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-05 11:43:20 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 11:43:20 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 11:43:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 11:43:20 0 [Note] InnoDB: Uses event mutexes 2020-08-05 11:43:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 11:43:20 0 [Note] InnoDB: Number of pools: 1 2020-08-05 11:43:20 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 11:43:20 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 11:43:23 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 11:43:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 11:43:23 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=679893622 2020-08-05 11:43:24 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000154', position 77312 2020-08-05 11:43:24 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 11:43:24 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 11:43:24 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 11:43:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 11:43:24 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 11:43:24 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 11:43:24 0 [Note] InnoDB: 10.3.23 started; log sequence number 679893631; transaction id 26911 2020-08-05 11:43:24 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 11:43:24 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 11:43:24 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 11:43:24 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 11:43:24 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 11:43:24 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 11:43:24 0 [Note] Starting ack receiver thread 2020-08-05 11:43:24 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-05 11:43:24 0 [Note] Starting crash recovery... 2020-08-05 11:43:24 0 [Note] Crash recovery finished. 2020-08-05 11:43:25 0 [Warning] Failed to setup SSL 2020-08-05 11:43:25 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 11:43:25 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 11:43:25 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 11:43:25 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 11:43:25 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 11:43:25 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 11:43:32 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 11:43:32 2020-08-05 11:56:38 157 [Warning] Timeout waiting for reply of binlog (file: bin_log.000155, pos: 1006), semi-sync up to file , position 0. 2020-08-05 11:56:38 157 [Note] Semi-sync replication switched OFF. 2020-08-05 12:07:21 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-08-05 12:07:21 0 [Note] InnoDB: FTS optimize thread exiting. 2020-08-05 12:07:21 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-08-05 12:07:21 6 [Note] Stopping ack receiver thread 200805 12:07:22 server_audit: STOPPED 2020-08-05 12:07:24 0 [Note] InnoDB: Starting shutdown... 2020-08-05 12:07:24 0 [Note] InnoDB: Dumping buffer pool(s) to /mariadb/data/ib_buffer_pool 2020-08-05 12:07:24 0 [Note] InnoDB: Buffer pool(s) dump completed at 200805 12:07:24 2020-08-05 12:07:29 0 [Note] InnoDB: Shutdown completed; log sequence number 679909690; transaction id 26973 2020-08-05 12:07:29 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 12:07:29 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-08-05 12:07:53 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 12:07:53 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 12:07:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 12:07:53 0 [Note] InnoDB: Uses event mutexes 2020-08-05 12:07:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 12:07:53 0 [Note] InnoDB: Number of pools: 1 2020-08-05 12:07:53 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 12:07:53 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 12:07:56 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 12:07:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 12:07:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 12:07:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 12:07:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 12:07:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 12:07:58 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 12:07:58 0 [Note] InnoDB: 10.3.23 started; log sequence number 679909690; transaction id 26975 2020-08-05 12:07:58 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 12:07:58 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 12:07:58 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 12:07:58 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 12:07:58 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 12:07:58 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 12:07:58 0 [Note] Starting ack receiver thread 2020-08-05 12:08:02 0 [Warning] Failed to setup SSL 2020-08-05 12:08:02 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 12:08:02 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 12:08:02 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 12:08:02 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 12:08:02 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 12:08:02 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 12:08:07 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 12:08:07 2020-08-05 12:11:54 87 [Warning] Timeout waiting for reply of binlog (file: bin_log.000156, pos: 1006), semi-sync up to file , position 0. 2020-08-05 12:11:54 87 [Note] Semi-sync replication switched OFF. 2020-08-05 12:20:16 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown 2020-08-05 12:20:16 0 [Note] InnoDB: FTS optimize thread exiting. 2020-08-05 12:20:16 0 [Note] Event Scheduler: Purging the queue. 0 events 2020-08-05 12:20:16 6 [Note] Stopping ack receiver thread 200805 12:20:16 server_audit: STOPPED 2020-08-05 12:20:17 0 [Note] InnoDB: Starting shutdown... 2020-08-05 12:20:17 0 [Note] InnoDB: Dumping buffer pool(s) to /mariadb/data/ib_buffer_pool 2020-08-05 12:20:17 0 [Note] InnoDB: Buffer pool(s) dump completed at 200805 12:20:17 2020-08-05 12:20:23 0 [Note] InnoDB: Shutdown completed; log sequence number 679926828; transaction id 27037 2020-08-05 12:20:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 12:20:23 0 [Note] /usr/sbin/mysqld: Shutdown complete 2020-08-05 12:20:37 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 12:20:37 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 12:20:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 12:20:37 0 [Note] InnoDB: Uses event mutexes 2020-08-05 12:20:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 12:20:37 0 [Note] InnoDB: Number of pools: 1 2020-08-05 12:20:37 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 12:20:37 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 12:20:41 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 12:20:41 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 12:20:43 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 12:20:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 12:20:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 12:20:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 12:20:43 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 12:20:43 0 [Note] InnoDB: 10.3.23 started; log sequence number 679926828; transaction id 27039 2020-08-05 12:20:43 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 12:20:43 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 12:20:43 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 12:20:43 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 12:20:43 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 12:20:43 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 12:20:43 0 [Note] Starting ack receiver thread 2020-08-05 12:20:45 0 [Warning] Failed to setup SSL 2020-08-05 12:20:45 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 12:20:45 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 12:20:45 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 12:20:45 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 12:20:45 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 12:20:45 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 12:20:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 12:20:50 2020-08-05 12:28:26 133 [Warning] Timeout waiting for reply of binlog (file: bin_log.000157, pos: 1006), semi-sync up to file , position 0. 2020-08-05 12:28:26 133 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200805 18:13:58 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=94 max_threads=1002 thread_count=100 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f75a00becb0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f75dc9b2d58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x5560af5fb309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5560aeebe67f] sigaction.c:0(__restore_rt)[0x7f7b79b75630] :0(__GI_raise)[0x7f7b77a2a377] :0(__GI_abort)[0x7f7b77a2ba68] :0(__assert_fail_base)[0x7f7b77a23196] :0(__GI___assert_fail)[0x7f7b77a23242] /usr/sbin/mysqld(+0xeaef93)[0x5560af4eaf93] /usr/sbin/mysqld(+0xeb1b2c)[0x5560af4edb2c] /usr/sbin/mysqld(+0xeb454f)[0x5560af4f054f] /usr/sbin/mysqld(+0xea904c)[0x5560af4e504c] /usr/sbin/mysqld(+0xeaae74)[0x5560af4e6e74] /usr/sbin/mysqld(+0xeb7805)[0x5560af4f3805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x5560aeceb4c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x5560aecf52ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x5560aecf593f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x5560aecf99e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x5560aecdcffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x5560aecdd8b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x5560aecdb829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x5560aecdc333] /usr/sbin/mysqld(+0x4f5135)[0x5560aeb31135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x5560aec74fb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x5560aec7bcea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x5560aec7d9f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x5560aec7ffef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x5560aed82077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x5560aed82134] pthread_create.c:0(start_thread)[0x7f7b79b6dea5] /lib64/libc.so.6(clone+0x6d)[0x7f7b77af28cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f75a00482d8): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 3813 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-05 18:22:09 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 18:22:09 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 18:22:09 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 18:22:09 0 [Note] InnoDB: Uses event mutexes 2020-08-05 18:22:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 18:22:09 0 [Note] InnoDB: Number of pools: 1 2020-08-05 18:22:09 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 18:22:09 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 18:22:12 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 18:22:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 18:22:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=680343185 2020-08-05 18:22:15 0 [Note] InnoDB: Transaction 28577 was in the XA prepared state. 2020-08-05 18:22:15 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo 2020-08-05 18:22:15 0 [Note] InnoDB: Trx id counter is 28578 2020-08-05 18:22:15 0 [Note] InnoDB: Starting final batch to recover 1 pages from redo log. 2020-08-05 18:22:16 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000157', position 440237 2020-08-05 18:22:16 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 18:22:16 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 2020-08-05 18:22:16 0 [Note] InnoDB: Rollback of non-prepared transactions completed 2020-08-05 18:22:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 18:22:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 18:22:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 18:22:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 18:22:16 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 18:22:16 0 [Note] InnoDB: 10.3.23 started; log sequence number 680343239; transaction id 28580 2020-08-05 18:22:16 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 18:22:16 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 18:22:16 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 18:22:16 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 18:22:16 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 18:22:16 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 18:22:16 0 [Note] Starting ack receiver thread 2020-08-05 18:22:16 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-05 18:22:16 0 [Note] Starting crash recovery... 2020-08-05 18:22:16 0 [Note] InnoDB: Starting recovery for XA transactions... 2020-08-05 18:22:16 0 [Note] InnoDB: Transaction 28577 in prepared state after recovery 2020-08-05 18:22:16 0 [Note] InnoDB: Transaction contains changes to 1 rows 2020-08-05 18:22:16 0 [Note] InnoDB: 1 transactions in prepared state after recovery 2020-08-05 18:22:16 0 [Note] Found 1 prepared transaction(s) in InnoDB 2020-08-05 18:22:17 0 [Note] Crash recovery finished. 2020-08-05 18:24:12 0 [Warning] Failed to setup SSL 2020-08-05 18:24:12 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 18:24:12 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 18:24:12 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 18:24:12 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 18:24:12 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 18:24:27 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 18:30:22 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 18:30:22 2020-08-05 18:30:57 13 [Warning] Timeout waiting for reply of binlog (file: bin_log.000158, pos: 1006), semi-sync up to file , position 0. 2020-08-05 18:30:57 13 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200805 20:23:21 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=106 max_threads=1002 thread_count=114 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7fce8004ea50 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fcec9ef2d58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x55cc53df3309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55cc536b667f] sigaction.c:0(__restore_rt)[0x7fd44fc2d630] :0(__GI_raise)[0x7fd44dae2377] :0(__GI_abort)[0x7fd44dae3a68] :0(__assert_fail_base)[0x7fd44dadb196] :0(__GI___assert_fail)[0x7fd44dadb242] /usr/sbin/mysqld(+0xeaef93)[0x55cc53ce2f93] /usr/sbin/mysqld(+0xeb1b2c)[0x55cc53ce5b2c] /usr/sbin/mysqld(+0xeb454f)[0x55cc53ce854f] /usr/sbin/mysqld(+0xea904c)[0x55cc53cdd04c] /usr/sbin/mysqld(+0xeaae74)[0x55cc53cdee74] /usr/sbin/mysqld(+0xeb7805)[0x55cc53ceb805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x55cc534e34c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x55cc534ed2ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x55cc534ed93f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x55cc534f19e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x55cc534d4ffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x55cc534d58b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x55cc534d3829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x55cc534d4333] /usr/sbin/mysqld(+0x4f5135)[0x55cc53329135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x55cc5346cfb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x55cc53473cea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x55cc534759f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x55cc53477fef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x55cc5357a077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x55cc5357a134] pthread_create.c:0(start_thread)[0x7fd44fc25ea5] /lib64/libc.so.6(clone+0x6d)[0x7fd44dbaa8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fce8005fbe8): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 1301 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-05 20:29:23 0 [Note] InnoDB: Using Linux native AIO 2020-08-05 20:29:23 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-05 20:29:23 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-05 20:29:23 0 [Note] InnoDB: Uses event mutexes 2020-08-05 20:29:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-05 20:29:23 0 [Note] InnoDB: Number of pools: 1 2020-08-05 20:29:23 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-05 20:29:23 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-05 20:29:27 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-05 20:29:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-05 20:29:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=680669467 2020-08-05 20:29:35 0 [Note] InnoDB: Transaction 29804 was in the XA prepared state. 2020-08-05 20:29:35 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo 2020-08-05 20:29:35 0 [Note] InnoDB: Trx id counter is 29805 2020-08-05 20:29:35 0 [Note] InnoDB: Starting final batch to recover 41 pages from redo log. 2020-08-05 20:29:36 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000158', position 341942 2020-08-05 20:29:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-05 20:29:36 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 2020-08-05 20:29:36 0 [Note] InnoDB: Rollback of non-prepared transactions completed 2020-08-05 20:29:45 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-05 20:29:45 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-05 20:29:45 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-05 20:29:45 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-05 20:29:45 0 [Note] InnoDB: Waiting for purge to start 2020-08-05 20:29:45 0 [Note] InnoDB: 10.3.23 started; log sequence number 680686121; transaction id 29807 2020-08-05 20:29:45 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-05 20:29:45 0 [Note] Plugin 'FEEDBACK' is disabled. 200805 20:29:45 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200805 20:29:45 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-05 20:29:45 0 [Note] Semi-sync replication initialized for transactions. 2020-08-05 20:29:45 0 [Note] Semi-sync replication enabled on the master. 2020-08-05 20:29:45 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-05 20:29:45 0 [Note] Starting ack receiver thread 2020-08-05 20:29:45 0 [Note] Starting crash recovery... 2020-08-05 20:29:45 0 [Note] InnoDB: Starting recovery for XA transactions... 2020-08-05 20:29:45 0 [Note] InnoDB: Transaction 29804 in prepared state after recovery 2020-08-05 20:29:45 0 [Note] InnoDB: Transaction contains changes to 1 rows 2020-08-05 20:29:45 0 [Note] InnoDB: 1 transactions in prepared state after recovery 2020-08-05 20:29:45 0 [Note] Found 1 prepared transaction(s) in InnoDB 2020-08-05 20:29:52 0 [Note] InnoDB: Buffer pool(s) load completed at 200805 20:29:52 2020-08-05 20:29:58 0 [Note] Crash recovery finished. 2020-08-05 20:30:33 0 [Warning] Failed to setup SSL 2020-08-05 20:30:33 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-05 20:30:33 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-05 20:30:33 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-05 20:30:33 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-05 20:30:33 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-05 20:30:33 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-05 20:37:14 10 [Warning] Timeout waiting for reply of binlog (file: bin_log.000159, pos: 1006), semi-sync up to file , position 0. 2020-08-05 20:37:14 10 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200806 17:54:12 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=93 max_threads=1002 thread_count=101 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f71f00858a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f7265739d58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x55ea4360f309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55ea42ed267f] sigaction.c:0(__restore_rt)[0x7f77ec3fe630] :0(__GI_raise)[0x7f77ea2b3377] :0(__GI_abort)[0x7f77ea2b4a68] :0(__assert_fail_base)[0x7f77ea2ac196] :0(__GI___assert_fail)[0x7f77ea2ac242] /usr/sbin/mysqld(+0xeaef93)[0x55ea434fef93] /usr/sbin/mysqld(+0xeb1b2c)[0x55ea43501b2c] /usr/sbin/mysqld(+0xeb454f)[0x55ea4350454f] /usr/sbin/mysqld(+0xea904c)[0x55ea434f904c] /usr/sbin/mysqld(+0xeaae74)[0x55ea434fae74] /usr/sbin/mysqld(+0xeb7805)[0x55ea43507805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x55ea42cff4c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x55ea42d092ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x55ea42d0993f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x55ea42d0d9e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x55ea42cf0ffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x55ea42cf18b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x55ea42cef829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x55ea42cf0333] /usr/sbin/mysqld(+0x4f5135)[0x55ea42b45135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x55ea42c88fb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x55ea42c8fcea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x55ea42c919f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x55ea42c93fef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x55ea42d96077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x55ea42d96134] pthread_create.c:0(start_thread)[0x7f77ec3f6ea5] /lib64/libc.so.6(clone+0x6d)[0x7f77ea37b8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f71f00ea548): SHOW PROCEDURE STATUS WHERE `Db`='ofcrmid' Connection ID (thread ID): 13752 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-06 17:54:45 0 [Note] InnoDB: Using Linux native AIO 2020-08-06 17:54:45 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-06 17:54:45 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-06 17:54:45 0 [Note] InnoDB: Uses event mutexes 2020-08-06 17:54:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-06 17:54:45 0 [Note] InnoDB: Number of pools: 1 2020-08-06 17:54:45 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-06 17:54:45 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-06 17:54:48 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-06 17:54:49 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-06 17:54:49 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=683018776 2020-08-06 17:54:49 0 [Note] InnoDB: Starting final batch to recover 5 pages from redo log. 2020-08-06 17:54:50 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000161', position 766567 2020-08-06 17:54:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-06 17:54:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-06 17:54:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-06 17:54:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-06 17:54:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-06 17:54:50 0 [Note] InnoDB: Waiting for purge to start 2020-08-06 17:54:50 0 [Note] InnoDB: 10.3.23 started; log sequence number 683018785; transaction id 35913 2020-08-06 17:54:50 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-06 17:54:50 0 [Note] Plugin 'FEEDBACK' is disabled. 200806 17:54:50 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200806 17:54:50 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-06 17:54:50 0 [Note] Semi-sync replication initialized for transactions. 2020-08-06 17:54:50 0 [Note] Semi-sync replication enabled on the master. 2020-08-06 17:54:50 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-06 17:54:50 0 [Note] Starting ack receiver thread 2020-08-06 17:54:50 0 [Note] Starting crash recovery... 2020-08-06 17:54:50 0 [Note] Crash recovery finished. 2020-08-06 17:54:51 0 [Warning] Failed to setup SSL 2020-08-06 17:54:51 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-06 17:54:51 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-06 17:54:51 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-06 17:54:51 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-06 17:54:51 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-06 17:54:51 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-06 17:55:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200806 17:55:00 2020-08-06 17:59:42 94 [Warning] Timeout waiting for reply of binlog (file: bin_log.000162, pos: 1006), semi-sync up to file , position 0. 2020-08-06 17:59:42 94 [Note] Semi-sync replication switched OFF. mysqld: /home/abychko/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX_ON_ES_BACKUP_DEBUGSOURCE_one_two_three_four_five/storage/maria/ma_bitmap.c:3008: _ma_check_bitmap: Assertion `(bitmap->map + (bitmap->full_head_size/6*6)) <= full_head_end' failed. 200806 21:00:22 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.23-MariaDB-debug-log key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=105 max_threads=1002 thread_count=113 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2268997 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f1f54037dd0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1fd741fd58 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x55985098a309] /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55985024d67f] sigaction.c:0(__restore_rt)[0x7f2556962630] :0(__GI_raise)[0x7f2554817377] :0(__GI_abort)[0x7f2554818a68] :0(__assert_fail_base)[0x7f2554810196] :0(__GI___assert_fail)[0x7f2554810242] /usr/sbin/mysqld(+0xeaef93)[0x559850879f93] /usr/sbin/mysqld(+0xeb1b2c)[0x55985087cb2c] /usr/sbin/mysqld(+0xeb454f)[0x55985087f54f] /usr/sbin/mysqld(+0xea904c)[0x55985087404c] /usr/sbin/mysqld(+0xeaae74)[0x559850875e74] /usr/sbin/mysqld(+0xeb7805)[0x559850882805] /usr/sbin/mysqld(_Z25schema_table_store_recordP3THDP5TABLE+0x188)[0x55985007a4c8] /usr/sbin/mysqld(_Z17store_schema_procP3THDP5TABLES2_PKcbS4_+0x5bd)[0x5598500842ad] /usr/sbin/mysqld(_Z16fill_schema_procP3THDP10TABLE_LISTP4Item+0x48f)[0x55985008493f] /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f2)[0x5598500889e2] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x86d)[0x55985006bffd] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x48)[0x55985006c8b8] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x129)[0x55985006a829] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x233)[0x55985006b333] /usr/sbin/mysqld(+0x4f5135)[0x55984fec0135] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x42c8)[0x559850003fb8] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x43a)[0x55985000acea] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1232)[0x55985000c9f2] /usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0x55985000efef] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x247)[0x559850111077] /usr/sbin/mysqld(handle_one_connection+0x34)[0x559850111134] pthread_create.c:0(start_thread)[0x7f255695aea5] /lib64/libc.so.6(clone+0x6d)[0x7f25548df8cd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f1f54048ca8): SHOW PROCEDURE STATUS WHERE Db = 'batchcrmid' Connection ID (thread ID): 2035 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /mariadb/d... Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 127956 127956 processes Max open files 65536 65536 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 127956 127956 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: co... 2020-08-06 21:00:52 0 [Note] InnoDB: Using Linux native AIO 2020-08-06 21:00:52 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2020-08-06 21:00:52 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2020-08-06 21:00:52 0 [Note] InnoDB: Uses event mutexes 2020-08-06 21:00:52 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 2020-08-06 21:00:52 0 [Note] InnoDB: Number of pools: 1 2020-08-06 21:00:52 0 [Note] InnoDB: Using SSE2 crc32 instructions 2020-08-06 21:00:52 0 [Note] InnoDB: Initializing buffer pool, total size = 20G, instances = 8, chunk size = 128M 2020-08-06 21:00:55 0 [Note] InnoDB: Completed initialization of buffer pool 2020-08-06 21:00:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2020-08-06 21:00:55 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=683475650 2020-08-06 21:00:56 0 [Note] InnoDB: Last binlog file '/mariadb/bin/logs/bin_logs/bin_log.000162', position 474737 2020-08-06 21:00:56 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2020-08-06 21:00:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2020-08-06 21:00:56 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2020-08-06 21:00:56 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2020-08-06 21:00:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2020-08-06 21:00:56 0 [Note] InnoDB: Waiting for purge to start 2020-08-06 21:00:56 0 [Note] InnoDB: 10.3.23 started; log sequence number 683475659; transaction id 37076 2020-08-06 21:00:56 0 [Note] InnoDB: Loading buffer pool(s) from /mariadb/data/ib_buffer_pool 2020-08-06 21:00:56 0 [Note] Plugin 'FEEDBACK' is disabled. 200806 21:00:56 server_audit: MariaDB Audit Plugin version 1.4.8-debug STARTED. 200806 21:00:56 server_audit: logging started to the file /mariadb/bin/logs/audit_logs/audit_mariadb.log. 2020-08-06 21:00:56 0 [Note] Semi-sync replication initialized for transactions. 2020-08-06 21:00:56 0 [Note] Semi-sync replication enabled on the master. 2020-08-06 21:00:56 0 [Note] Recovering after a crash using /mariadb/bin/logs/bin_logs/bin_log 2020-08-06 21:00:56 0 [Note] Starting ack receiver thread 2020-08-06 21:00:56 0 [Note] Starting crash recovery... 2020-08-06 21:00:56 0 [Note] Crash recovery finished. 2020-08-06 21:00:56 0 [Warning] Failed to setup SSL 2020-08-06 21:00:56 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2020-08-06 21:00:56 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory 2020-08-06 21:00:56 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file 2020-08-06 21:00:56 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib 2020-08-06 21:00:56 0 [Note] Server socket created on IP: '10.91.194.126'. 2020-08-06 21:00:56 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.23-MariaDB-debug-log' socket: '/var/lib/mysql/mysql.sock' port: 6603 Source distribution 2020-08-06 21:01:04 0 [Note] InnoDB: Buffer pool(s) load completed at 200806 21:01:04 2020-08-06 21:15:05 192 [Warning] Timeout waiting for reply of binlog (file: bin_log.000163, pos: 1006), semi-sync up to file , position 0. 2020-08-06 21:15:05 192 [Note] Semi-sync replication switched OFF.