Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.0.24, 10.1(EOL)
-
Debian Wheezy/Jessie, 64bit.
Description
MariaDB suddenly crashed, attaching log file. We really didn't mix up frm files.
pr 29 15:01:28 server02 mysqld: InnoDB: Warning: Index PRIMARY points to table jsworks_iqcrmsk/stranka_moduly and ib_table jsworks_iqcrmsk/stranka_moduly statistics is initialized 1 but index table jsworks_iqcrmsk/stranka_moduly initialized 0 mysql table is stranka_moduly. Have you mixed up .frm files from different installations? See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html
|
Apr 29 15:01:29 server02 mysqld: 2016-04-29 15:01:29 7fabb9175700 InnoDB: Assertion failure in thread 140375521449728 in file ha_innodb.cc line 7067
|
Apr 29 15:01:29 server02 mysqld: InnoDB: Failing assertion: templ->clust_rec_field_no != ULINT_UNDEFINED
|
Apr 29 15:01:29 server02 mysqld: InnoDB: We intentionally generate a memory trap.
|
Apr 29 15:01:29 server02 mysqld: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
|
Apr 29 15:01:29 server02 mysqld: InnoDB: If you get repeated assertion failures or crashes, even
|
Apr 29 15:01:29 server02 mysqld: InnoDB: immediately after the mysqld startup, there may be
|
Apr 29 15:01:29 server02 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Apr 29 15:01:29 server02 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
|
Apr 29 15:01:29 server02 mysqld: InnoDB: about forcing recovery.
|
Apr 29 15:01:29 server02 mysqld: 160429 15:01:29 [ERROR] mysqld got signal 6 ;
|
Apr 29 15:01:29 server02 mysqld: This could be because you hit a bug. It is also possible that this binary
|
Apr 29 15:01:29 server02 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Apr 29 15:01:29 server02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Apr 29 15:01:29 server02 mysqld:
|
Apr 29 15:01:29 server02 mysqld: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
|
Apr 29 15:01:29 server02 mysqld:
|
Apr 29 15:01:29 server02 mysqld: We will try our best to scrape up some info that will hopefully help
|
Apr 29 15:01:29 server02 mysqld: diagnose the problem, but since we have already crashed,
|
Apr 29 15:01:29 server02 mysqld: something is definitely wrong and this may fail.
|
Apr 29 15:01:29 server02 mysqld:
|
Apr 29 15:01:29 server02 mysqld: Server version: 10.0.24-MariaDB-1~wheezy
|
Apr 29 15:01:29 server02 mysqld: key_buffer_size=2147483648
|
Apr 29 15:01:29 server02 mysqld: read_buffer_size=131072
|
Apr 29 15:01:29 server02 mysqld: max_used_connections=151
|
Apr 29 15:01:29 server02 mysqld: max_threads=152
|
Apr 29 15:01:29 server02 mysqld: thread_count=30
|
Apr 29 15:01:29 server02 mysqld: It is possible that mysqld could use up to
|
Apr 29 15:01:29 server02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2430889 K bytes of memory
|
Apr 29 15:01:29 server02 mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Apr 29 15:01:29 server02 mysqld:
|
Apr 29 15:01:29 server02 mysqld: Thread pointer: 0x0x7faacc5a0008
|
Apr 29 15:01:29 server02 mysqld: Attempting backtrace. You can use the following information to find out
|
Apr 29 15:01:29 server02 mysqld: where mysqld died. If you see no messages after this, something went
|
Apr 29 15:01:29 server02 mysqld: terribly wrong...
|
Apr 29 15:01:29 server02 mysqld: stack_bottom = 0x7fabb9174e10 thread_stack 0x40000
|
Apr 29 15:01:29 server02 kernel: [17834753.884677] grsec: From 127.0.0.6: failed fork with errno ENOMEM by /usr/sbin/mysqld[mysqld:27488] uid/euid:105/105 gid/egid:109/109, parent /usr/bin/mysqld_safe[mysqld_safe:18591] uid/euid:0/0 gid/egid:0/0
|
Apr 29 15:01:29 server02 mysqld: (my_addr_resolve failure: fork)
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2b) [0x7fac7c0a24cb]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x422) [0x7fac7bc280e2]
|
Apr 29 15:01:30 server02 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0) [0x7fac7b2700a0]
|
Apr 29 15:01:30 server02 mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fac798c9125]
|
Apr 29 15:01:30 server02 mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x180) [0x7fac798cc3a0]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x7144f0) [0x7fac7bdb34f0]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x71fed0) [0x7fac7bdbeed0]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x722729) [0x7fac7bdc1729]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x72296b) [0x7fac7bdc196b]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(handler::ha_rnd_init_with_error(bool)+0x19) [0x7fac7bc2d989]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(init_read_record(READ_RECORD*, THD*, TABLE*, SQL_SELECT*, int, bool, bool)+0x4f6) [0x7fac7bd1f656]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(join_init_read_record(st_join_table*)+0x80) [0x7fac7bafac70]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x161) [0x7fac7baedf81]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x466a0d) [0x7fac7bb05a0d]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(JOIN::exec_inner()+0xa6c) [0x7fac7bb1903c]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(JOIN::exec()+0x11) [0x7fac7bb1ae11]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1ad) [0x7fac7bb17a7d]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(handle_select(THD*, LEX*, select_result*, unsigned long)+0x28d) [0x7fac7bb1b16d]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x420511) [0x7fac7babf511]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(mysql_execute_command(THD*)+0x4ddc) [0x7fac7bacaa4c]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(+0x42d583) [0x7fac7bacc583]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1d64) [0x7fac7baceae4]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x47b) [0x7fac7bb918ab]
|
Apr 29 15:01:30 server02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x47) [0x7fac7bb91987]
|
Apr 29 15:01:30 server02 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7fac7b267b50]
|
Apr 29 15:01:30 server02 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fac7997530d]
|
Apr 29 15:01:30 server02 mysqld:
|
Apr 29 15:01:30 server02 mysqld: Trying to get some variables.
|
Apr 29 15:01:30 server02 mysqld: Some pointers may be invalid and cause the dump to abort.
|
Apr 29 15:01:30 server02 mysqld: Query (0x7fa889304020): is an invalid pointer
|
Apr 29 15:01:30 server02 mysqld: Connection ID (thread ID): 8740039
|
Apr 29 15:01:30 server02 mysqld: Status: NOT_KILLED
|
Apr 29 15:01:30 server02 mysqld:
|
Apr 29 15:01:30 server02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
|
Attachments
Issue Links
- relates to
-
-
- Closed
-
Activity
As discussed on Slack, I might try to use this injection to force table eviction:
DBUG_EXECUTE_IF("innodb_evict_autoinc_table",
|
mutex_enter(&dict_sys->mutex);
|
dict_table_remove_from_cache_low(user_table, TRUE);
|
mutex_exit(&dict_sys->mutex);
|
);
|
In MariaDB 10.3.18 and 10.4.8, we introduced innodb_evict_tables_on_commit_debug, which could be set to trigger these bugs. But, I remember that mleich reported some bugs related to this instrumentation itself.
Can anyone reproduce this bug? We have not seen this in our internal testing. As far as I can tell, the MDEV-6424 changes were never reverted despite jplindst’s intention to do so.
Perhaps mleich should enable innodb_evict_tables_on_commit_debug in his tests, now that the bugs in its implementation were fixed in MDEV-15880.
I have a hypothesis that this could be related to table eviction. InnoDB in MySQL 5.6 (and in MariaDB 10.x) implemented eviction of least recently used table definitions from the dict_sys cache.
I think that the problem should occur when a table definition has been first opened internally in InnoDB, and then from SQL. This either has to occur right after server startup, or some time after table definition eviction. Note that the FLUSH TABLES statement is not directly connected to the InnoDB table cache eviction.
I would like to get access to a core dump that demonstrates this problem. Possibly we might have to revert the
MDEV-6424change so that the crash occurs sooner after the inconsistency has been detected, and we would have better stack traces of the ‘conflicting threads’.