Dec 10 21:15:42 dbsrv mysqld: 2020-12-10 21:15:42 891153 [ERROR] [FATAL] InnoDB: Unknown error Required history data has been deleted Dec 10 21:15:42 dbsrv mysqld: 201210 21:15:42 [ERROR] mysqld got signal 6 ; Dec 10 21:15:42 dbsrv mysqld: This could be because you hit a bug. It is also possible that this binary Dec 10 21:15:42 dbsrv mysqld: or one of the libraries it was linked against is corrupt, improperly built, Dec 10 21:15:42 dbsrv mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Dec 10 21:15:42 dbsrv mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Dec 10 21:15:42 dbsrv mysqld: We will try our best to scrape up some info that will hopefully help Dec 10 21:15:42 dbsrv mysqld: diagnose the problem, but since we have already crashed, Dec 10 21:15:42 dbsrv mysqld: something is definitely wrong and this may fail. Dec 10 21:15:42 dbsrv mysqld: Server version: 10.3.27-MariaDB-log Dec 10 21:15:42 dbsrv mysqld: key_buffer_size=134217728 Dec 10 21:15:42 dbsrv mysqld: read_buffer_size=131072 Dec 10 21:15:43 dbsrv mysqld: max_used_connections=82 Dec 10 21:15:43 dbsrv mysqld: max_threads=302 Dec 10 21:15:43 dbsrv mysqld: thread_count=98 Dec 10 21:15:43 dbsrv mysqld: It is possible that mysqld could use up to Dec 10 21:15:43 dbsrv mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 795005 K bytes of memory Dec 10 21:15:43 dbsrv mysqld: Hope that's ok; if not, decrease some variables in the equation. Dec 10 21:15:43 dbsrv mysqld: Thread pointer: 0x7fbc100009a8 Dec 10 21:15:43 dbsrv mysqld: Attempting backtrace. You can use the following information to find out Dec 10 21:15:43 dbsrv mysqld: where mysqld died. If you see no messages after this, something went Dec 10 21:15:43 dbsrv mysqld: terribly wrong... Dec 10 21:15:43 dbsrv mysqld: stack_bottom = 0x7fbd390ebd30 thread_stack 0x49000 Dec 10 21:15:43 dbsrv mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55c2faf8d66e] Dec 10 21:15:44 dbsrv mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55c2faa24e7f] Dec 10 21:15:44 dbsrv mysqld: /lib64/libpthread.so.0(+0xf630)[0x7fc9dafb7630] Dec 10 21:15:44 dbsrv mysqld: /lib64/libc.so.6(gsignal+0x37)[0x7fc9d9287387] Dec 10 21:15:45 dbsrv mysqld: /lib64/libc.so.6(abort+0x148)[0x7fc9d9288a78] Dec 10 21:15:45 dbsrv mysqld: /usr/sbin/mysqld(+0xaa0b73)[0x55c2fad15b73] Dec 10 21:15:46 dbsrv mysqld: /usr/sbin/mysqld(+0xa15f17)[0x55c2fac8af17] Dec 10 21:15:46 dbsrv mysqld: /usr/sbin/mysqld(+0xa34c48)[0x55c2faca9c48] Dec 10 21:15:46 dbsrv mysqld: /usr/sbin/mysqld(+0x955877)[0x55c2fabca877] Dec 10 21:15:47 dbsrv mysqld: /usr/sbin/mysqld(_ZN7handler13ha_index_nextEPh+0x19f)[0x55c2faa2acbf] Dec 10 21:15:47 dbsrv mysqld: /usr/sbin/mysqld(_ZN7handler15read_range_nextEv+0x20)[0x55c2faa2f440] Dec 10 21:15:47 dbsrv mysqld: /usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPv+0xb2)[0x55c2fa951c22] Dec 10 21:15:48 dbsrv mysqld: /usr/sbin/mysqld(_ZN23Mrr_simple_index_reader8get_nextEPPv+0x20)[0x55c2fa951c90] Dec 10 21:15:48 dbsrv mysqld: /usr/sbin/mysqld(_ZN10DsMrr_impl10dsmrr_nextEPPv+0x42)[0x55c2fa952f02] Dec 10 21:15:49 dbsrv mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x99)[0x55c2fab355a9] Dec 10 21:15:49 dbsrv mysqld: /usr/sbin/mysqld(+0x8dfdb9)[0x55c2fab54db9] Dec 10 21:15:49 dbsrv mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x211)[0x55c2fa87d091] Dec 10 21:15:50 dbsrv mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa8b)[0x55c2fa89fffb] Dec 10 21:15:50 dbsrv mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55c2fa8a0213] Dec 10 21:15:51 dbsrv mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55c2fa8a036a] Dec 10 21:15:51 dbsrv mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1dc)[0x55c2fa8a0e8c] Dec 10 21:15:52 dbsrv mysqld: /usr/sbin/mysqld(+0x4dc0f9)[0x55c2fa7510f9] Dec 10 21:15:52 dbsrv mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x21bc)[0x55c2fa846c6c] Dec 10 21:15:53 dbsrv mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1fb)[0x55c2fa84d6ab] Dec 10 21:15:53 dbsrv mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xf76)[0x55c2fa84ef26] Dec 10 21:15:53 dbsrv mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11b)[0x55c2fa850adb] Dec 10 21:15:54 dbsrv mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1d6)[0x55c2fa926706] Dec 10 21:15:54 dbsrv mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55c2fa92681d] Dec 10 21:15:55 dbsrv mysqld: /lib64/libpthread.so.0(+0x7ea5)[0x7fc9dafafea5] Dec 10 21:15:55 dbsrv mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fc9d934f96d] Dec 10 21:15:55 dbsrv mysqld: Trying to get some variables. Dec 10 21:15:55 dbsrv mysqld: Some pointers may be invalid and cause the dump to abort. Dec 10 21:15:55 dbsrv mysqld: Query (0x7fbc1000f4a0): SELECT count(id) FROM jos__jobs WHERE process_id > 0 AND info='xxxxxx' AND name like 'xxxxxxx%' Dec 10 21:15:55 dbsrv mysqld: Connection ID (thread ID): 891153 Dec 10 21:15:55 dbsrv mysqld: Status: NOT_KILLED Dec 10 21:15:55 dbsrv mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Dec 10 21:15:55 dbsrv mysqld: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains Dec 10 21:15:55 dbsrv mysqld: information that should help you find out what is causing the crash. Dec 10 21:15:55 dbsrv mysqld: Writing a core file... Dec 10 21:15:55 dbsrv mysqld: Working directory at /app/mysql Dec 10 21:15:55 dbsrv mysqld: Resource Limits: Dec 10 21:15:55 dbsrv mysqld: Limit Soft Limit Hard Limit Units Dec 10 21:15:55 dbsrv mysqld: Max cpu time unlimited unlimited seconds Dec 10 21:15:55 dbsrv mysqld: Max file size unlimited unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max data size unlimited unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max stack size 8388608 unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max core file size 0 unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max resident set unlimited unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max processes 256972 256972 processes Dec 10 21:15:55 dbsrv mysqld: Max open files 65535 65535 files Dec 10 21:15:55 dbsrv mysqld: Max locked memory 65536 65536 bytes Dec 10 21:15:55 dbsrv mysqld: Max address space unlimited unlimited bytes Dec 10 21:15:55 dbsrv mysqld: Max file locks unlimited unlimited locks Dec 10 21:15:55 dbsrv mysqld: Max pending signals 256972 256972 signals Dec 10 21:15:55 dbsrv mysqld: Max msgqueue size 819200 819200 bytes Dec 10 21:15:55 dbsrv mysqld: Max nice priority 40 40 Dec 10 21:15:55 dbsrv mysqld: Max realtime priority 0 0 Dec 10 21:15:55 dbsrv mysqld: Max realtime timeout unlimited unlimited us Dec 10 21:15:55 dbsrv mysqld: Core pattern: core Dec 10 21:15:57 dbsrv systemd: mariadb.service: main process exited, code=killed, status=6/ABRT Dec 10 21:15:57 dbsrv systemd: Unit mariadb.service entered failed state. Dec 10 21:15:57 dbsrv systemd: mariadb.service failed. Dec 10 21:16:02 dbsrv systemd: mariadb.service holdoff time over, scheduling restart. Dec 10 21:16:02 dbsrv systemd: Stopped MariaDB 10.3.27 database server. Dec 10 21:16:02 dbsrv systemd: Starting MariaDB 10.3.27 database server... Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-log) starting as process 4165 ... Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Warning] Could not increase number of max_open_files to more than 65535 (request: 512339) Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Using Linux native AIO Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Uses event mutexes Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Number of pools: 1 Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Using SSE2 crc32 instructions Dec 10 21:16:02 dbsrv mysqld: 2020-12-10 21:16:02 0 [Note] InnoDB: Initializing buffer pool, total size = 48G, instances = 8, chunk size = 128M Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: Completed initialization of buffer pool Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: page_cleaner coordinator priority: -20 Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: page_cleaner worker priority: -20 Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: page_cleaner worker priority: -20 Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: page_cleaner worker priority: -20 Dec 10 21:16:04 dbsrv mysqld: 2020-12-10 21:16:04 0 [Note] InnoDB: page_cleaner worker priority: -20 Dec 10 21:16:05 dbsrv mysqld: 2020-12-10 21:16:05 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=35936690902297 Dec 10 21:16:05 dbsrv mysqld: 2020-12-10 21:16:05 0 [Note] InnoDB: Transaction 49050422561 was in the XA prepared state. Dec 10 21:16:05 dbsrv mysqld: 2020-12-10 21:16:05 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo Dec 10 21:16:05 dbsrv mysqld: 2020-12-10 21:16:05 0 [Note] InnoDB: Trx id counter is 49050422562 Dec 10 21:16:05 dbsrv mysqld: 2020-12-10 21:16:05 0 [Note] InnoDB: Starting final batch to recover 676 pages from redo log. Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Last binlog file './mariadb-db10-bin.207044', position 26276235 Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Starting in background the rollback of recovered transactions Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Rollback of non-prepared transactions completed Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Waiting for purge to start Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: 10.3.27 started; log sequence number 35936690905552; transaction id 49050422563 Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Loading buffer pool(s) from /app/mysql/ib_buffer_pool Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Plugin 'FEEDBACK' is disabled. Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Recovering after a crash using mariadb-db10-bin Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Starting crash recovery... Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Starting recovery for XA transactions... Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Transaction 49050422561 in prepared state after recovery Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: Transaction contains changes to 1 rows Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] InnoDB: 1 transactions in prepared state after recovery Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Found 1 prepared transaction(s) in InnoDB Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Crash recovery finished. Dec 10 21:16:06 dbsrv mysqld: 2020-12-10 21:16:06 0 [Note] Server socket created on IP: '::'.