[MDEV-23035] INNODB assertion failure -> crash Created: 2020-06-28  Updated: 2020-09-08  Resolved: 2020-09-06

Status: Closed
Project: MariaDB Server
Component/s: Optimizer, Storage Engine - InnoDB
Affects Version/s: 10.4.13
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Joseph Nahmias Assignee: Sergei Petrunia
Resolution: Incomplete Votes: 0
Labels: need_feedback
Environment:

Ubuntu 20.04 LTS

Issue Links:
Relates
relates to MDEV-16188 Use in-memory PK filters built from r... Closed

 Description    

InnoDB: Assertion failure in file /home/buildbot/buildbot/build/m
 
ariadb-10.4.13/storage/innobase/row/row0sel.cc line 3966
 
 
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
200628 16:30:52 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
Server version: 10.4.13-MariaDB-1:10.4.13+maria~focal-log
 
key_buffer_size=536870912
 
read_buffer_size=16777216
 
max_used_connections=10
 
max_threads=102
 
thread_count=50
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3033535 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7fcca4000c48
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fcdb2f13da8 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x32)[0x5572b70e1fa2]
 
/usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x5572b6b78cbd]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fd12343e3c0]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fd122b2d18b]
 
/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fd122b0c859]
 
/usr/sbin/mysqld(+0x599aaa)[0x5572b6858aaa]
 
/usr/sbin/mysqld(+0x58e7bc)[0x5572b684d7bc]
 
/usr/sbin/mysqld(+0xb72357)[0x5572b6e31357]
 
/usr/sbin/mysqld(+0xa8c116)[0x5572b6d4b116]
 
/usr/sbin/mysqld(_ZN7handler18ha_index_next_sameEPhPKhj+0x1da)[0x5572b6b80bfa]
 
/usr/sbin/mysqld(+0x6e6875)[0x5572b69a5875]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1d3)[0x5572b6997e33]
 
/usr/sbin/mysqld(+0x6c9009)[0x5572b6988009]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x210)[0x5572b6997e70]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xe0e)[0x5572b69c49ce]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x37)[0x5572b69c4c27]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0
 
x136)[0x5572b69c2d76]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x15a)[0x5572b69c375a]
 
ut/ut0rbt.cc:458(rbt_eject_node(ib_rbt_node_t*, ib_rbt_node_t*) [clone .part.0])[0x5572b694fca1]
 
row/row0sel.cc:2949(row_search_idx_cond_check(unsigned char*, row_prebuilt_t*, unsigned char const*, unsigned short const*) [clone .cold])[0x5572
 
b695de52]
 
row/row0sel.cc:5266(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x5572b69606e3]
 
handler/ha_innodb.cc:9543(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int))[0x5572b6961098]
 
sql/handler.cc:3027(handler::ha_index_next_same(unsigned char*, unsigned char const*, unsigned int))[0x5572b696487c]
 
sql/sql_select.cc:21193(join_read_next_same(READ_RECORD*))[0x5572b6964f12]
 
sql/records.h:69(READ_RECORD::read_record())[0x5572b6a5544e]
 
sql/sql_select.cc:19904(JOIN::exec_inner())[0x5572b6a5556f]
 
sql/sql_parse.cc:6359(execute_sqlcom_select(THD*, TABLE_LIST*))[0x5572b7090146]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7fd123432609]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fd122c09103]
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fcca4025e40): SELECT directory_prefix, s.project_id, s.project_id, s.key, s.value, s.type         FROM redcap_external_modules m
 
  JOIN redcap_external_module_settings s          ON m.external_module_id = s.external_module_id         WHERE (s.key IN ('version', 'enabled'))
 
Connection ID (thread ID): 3142
 
Status: KILL_CONNECTION
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,eng
 
ine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_
 
exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,out
 
er_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_
 
elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_push
 
down_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /data/mysql/data
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units
 
Max cpu time              unlimited            unlimited            seconds
 
Max file size             unlimited            unlimited            bytes
 
Max data size             unlimited            unlimited            bytes
 
Max stack size            8388608              unlimited            bytes
 
Max core file size        0                    unlimited            bytes
 
Max resident set          unlimited            unlimited            bytes
 
Max processes             62401                62401                processes
 
Max open files            16364                16364                files
 
Max locked memory         65536                65536                bytes
 
Max address space         unlimited            unlimited            bytes
 
Max file locks            unlimited            unlimited            locks
 
Max pending signals       62401                62401                signals
 
Max msgqueue size         819200               819200               bytes
 
Max nice priority         0                    0
 
Max realtime priority     0                    0
 
Max realtime timeout      unlimited            unlimited            us
 
Core pattern: |/usr/share/apport/apport %p %s %c %d %P ...
 
Jun 28 16:30:57 redcap-db02 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Jun 28 16:30:57 redcap-db02 systemd[1]: mariadb.service: Failed with result 'signal'.
 
Jun 28 16:31:02 redcap-db02 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
 
Jun 28 16:31:02 redcap-db02 systemd[1]: Stopped MariaDB 10.4.13 database server.



 Comments   
Comment by Marko Mäkelä [ 2020-08-04 ]

Joseph.Nahmias@mountsinai.org, can you give more detail, such as SHOW CREATE TABLE for the table? The query is already included.

The code in question was added by MDEV-16188. The crash is intentional, in the last ut_error line in row_search_idx_cond_check() that I quote below:

	switch (result) {
 
	case CHECK_POS:
 
	        if (handler_rowid_filter_is_active(prebuilt->pk_filter)) {
 
		        ut_ad(!prebuilt->index->is_primary());
 
		        if (prebuilt->clust_index_was_generated) {
 
                               ulint len;
 
                               dict_index_t* index = prebuilt->index;
 
                               const byte* data = rec_get_nth_field(
 
                                       rec, offsets, index->n_fields - 1,
 
                                       &len);
 
                               ut_ad(dict_index_get_nth_col(index,
 
                                                            index->n_fields - 1)
 
                                     ->prtype == (DATA_ROW_ID | DATA_NOT_NULL));
 
                               ut_ad(len == DATA_ROW_ID_LEN);
 
                               memcpy(prebuilt->row_id, data, DATA_ROW_ID_LEN);
 
                        }
 
                        result = handler_rowid_filter_check(prebuilt->pk_filter);
 
                        switch (result) {
 
                        case CHECK_NEG:
 
			        MONITOR_INC(MONITOR_ICP_NO_MATCH);
 
                                return(result);
 
                        case CHECK_OUT_OF_RANGE:
 
                                MONITOR_INC(MONITOR_ICP_OUT_OF_RANGE);
 
                                return(result);
 
                        case CHECK_POS:
 
                                break;
 
                        default:
 
                                ut_error;

Comment by Joseph Nahmias [ 2020-09-08 ]

Sorry, I did not notice your comment above until now. Here are the create table statements:

mysql> SHOW CREATE TABLE redcap_external_modules \G
 
*************************** 1. row ***************************
 
       Table: redcap_external_modules
 
Create Table: CREATE TABLE `redcap_external_modules` (
 
  `external_module_id` int(11) NOT NULL AUTO_INCREMENT,
 
  `directory_prefix` varchar(191) COLLATE utf8_unicode_ci NOT NULL,
 
  PRIMARY KEY (`external_module_id`),
 
  UNIQUE KEY `directory_prefix` (`directory_prefix`)
 
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
 
1 row in set (0.00 sec)
 
mysql> SHOW CREATE TABLE redcap_external_module_settings \G
 
*************************** 1. row ***************************
 
Table: redcap_external_module_settings
 
Create Table: CREATE TABLE `redcap_external_module_settings` (
 
  `external_module_id` int(11) NOT NULL,
 
  `project_id` int(11) DEFAULT NULL,
 
  `key` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
 
  `type` varchar(12) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'string',
 
  `value` mediumtext COLLATE utf8_unicode_ci NOT NULL,
 
  KEY `external_module_id` (`external_module_id`),
 
  KEY `key` (`key`),
 
  KEY `project_id` (`project_id`),
 
  KEY `value` (`value`(190)),
 
  CONSTRAINT `redcap_external_module_settings_ibfk_1` FOREIGN KEY (`external_module_id`) REFERENCES `redcap_external_modules` (`external_module_id`) ON DELETE CASCADE ON UPDATE CASCADE,
 
  CONSTRAINT `redcap_external_module_settings_ibfk_2` FOREIGN KEY (`project_id`) REFERENCES `redcap_projects` (`project_id`) ON DELETE CASCADE ON UPDATE CASCADE
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
 
1 row in set (0.00 sec)}}

Generated at Thu Feb 08 09:19:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.