Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.1.23, 10.0(EOL), 10.1(EOL), 10.2(EOL)
-
Debian 9, Server version: 10.1.23-MariaDB-9+deb9u1
-
10.2.10
Description
After a lot of activity (SELECTs and INSERTS for hours) with 100% CPU load of mysqld, the memory consumption grows and grows.
Then mysqld crashes and error.log says:
2017-08-03 19:51:53 139829468153600 [ERROR] InnoDB: InnoDB: Unable to allocate memory of size 8104.
|
|
|
2017-08-03 19:51:53 7f2c95c74700 InnoDB: Assertion failure in thread 139829468153600 in file ha_innodb.cc line 21990
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
|
InnoDB: about forcing recovery.
|
170803 19:51:53 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.1.23-MariaDB-9+deb9u1
|
key_buffer_size=16777216
|
read_buffer_size=131072
|
max_used_connections=9
|
max_threads=153
|
thread_count=2
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 78257 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f2bf46cc008
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f2c95c73cb8 thread_stack 0x30000
|
(my_addr_resolve failure: fork)
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e) [0x555d95eb39ce]
|
/usr/sbin/mysqld(handle_fatal_signal+0x3bd) [0x555d959fbd1d]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f2c959ab0c0]
|
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf) [0x7f2c942a5fcf]
|
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f2c942a73fa]
|
/usr/sbin/mysqld(+0x82009b) [0x555d95c8209b]
|
/usr/sbin/mysqld(+0x876fb7) [0x555d95cd8fb7]
|
/usr/sbin/mysqld(+0x9e2eed) [0x555d95e44eed]
|
/usr/sbin/mysqld(+0x9f07c6) [0x555d95e527c6]
|
/usr/sbin/mysqld(+0x8cb313) [0x555d95d2d313]
|
/usr/sbin/mysqld(+0x8228b5) [0x555d95c848b5]
|
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x37f) [0x555d95a05c5f]
|
/usr/sbin/mysqld(write_record(THD*, TABLE*, st_copy_info*)+0x238) [0x555d95851b38]
|
/usr/sbin/mysqld(select_insert::send_data(List<Item>&)+0xf1) [0x555d958523f1]
|
/usr/sbin/mysqld(+0x456813) [0x555d958b8813]
|
/usr/sbin/mysqld(+0x43a096) [0x555d9589c096]
|
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x12d) [0x555d958a51fd]
|
/usr/sbin/mysqld(+0x43a096) [0x555d9589c096]
|
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x15b) [0x555d958a522b]
|
/usr/sbin/mysqld(+0x453ebd) [0x555d958b5ebd]
|
/usr/sbin/mysqld(JOIN::exec_inner()+0xbe2) [0x555d958c8912]
|
/usr/sbin/mysqld(JOIN::exec()+0x54) [0x555d958ca5b4]
|
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0xe7) [0x555d958c6f47]
|
/usr/sbin/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x13f) [0x555d958c791f]
|
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x860b) [0x555d9587511b]
|
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x311) [0x555d95875a41]
|
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x2463) [0x555d95878dc3]
|
/usr/sbin/mysqld(do_command(THD*)+0x155) [0x555d958794d5]
|
/usr/sbin/mysqld(do_handle_one_connection(THD*)+0x1d2) [0x555d959435d2]
|
/usr/sbin/mysqld(handle_one_connection+0x40) [0x555d95943730]
|
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f2c959a1494]
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f2c9435baff]
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f2bf4020020): REPLACE INTO isys_search_idx (isys_search_idx__version, isys_search_idx__type, isys_search_idx__key, isys_search_idx__value, isys_search_idx__reference) SELECT 1, 'cmdb', CONCAT(isys_obj__isys_obj_type__id, '.custom_fields.', isys_catg_custom_fields_list__data__id, '.', REPLACE(isysgui_catg_custom__title, '.', '_'), '.', isysgui_catg_custom__id, '.', isys_catg_custom_fields_list__field_key), isys_catg_custom_fields_list__field_content, isys_obj__id FROM isys_catg_custom_fields_list INNER JOIN isys_obj ON isys_catg_custom_fields_list__isys_obj__id = isys_obj__id INNER JOIN isysgui_catg_custom ON isys_catg_custom_fields_list__isysgui_catg_custom__id = isysgui_catg_custom__id WHERE isys_catg_custom_fields_list__field_type IN ('f_text', 'f_textarea', 'f_link') AND isys_catg_custom_fields_list__isysgui_catg_custom__id = 9
|
Connection ID (thread ID): 11
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
|
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
2017-08-03 19:52:01 140411564941888 [Warning] option 'innodb-sort-buffer-size': unsigned value 134217728 adjusted to 67108864
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Using mutexes to ref count buffer pool pages
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: The InnoDB memory heap is disabled
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Compressed tables use zlib 1.2.8
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Using Linux native AIO
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Using SSE crc32 instructions
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Initializing buffer pool, size = 2.0G
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Completed initialization of buffer pool
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Highest supported file format is Barracuda.
|
2017-08-03 19:52:01 140411564941888 [Note] InnoDB: Starting crash recovery from checkpoint LSN=269528798739
|
2017-08-03 19:52:02 140411564941888 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
|