[MDEV-23192] Crash in row_search_mvcc() probably related to secondary index corruption Created: 2020-07-16  Updated: 2023-02-09  Resolved: 2021-05-31

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.4.13, 10.4.14
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Valerii Kravchuk Assignee: Valerii Kravchuk
Resolution: Incomplete Votes: 0
Labels: need_feedback

Issue Links:
Blocks
is blocked by MDEV-22924 Warning InnoDB: Index 'Marvão_idx3' c... Closed
Relates
relates to MDEV-23723 Crash when test_if_skip_sort_order() ... Closed
relates to MDEV-24449 Corruption of system tablespace or la... Closed
relates to MDEV-25442 Lateral derived optimization causes s... Closed

 Description   

Server crashes repeatedly with stack trace like this:

020-07-16  9:27:03 0 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
 
Version: '10.4.13-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 33306  MariaDB Server
 
200716  9:27:56 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.13-MariaDB-log
 
key_buffer_size=33554432
 
read_buffer_size=1048576
 
max_used_connections=74
 
max_threads=2010
 
thread_count=80
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10348902 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f28283308b8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f28aafb1de0 thread_stack 0x49000
 
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x55af7c963e2e]
 
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x30f)[0x55af7c374b3f]
 
sigaction.c:0(__restore_rt)[0x7f2d914245e0]
 
/usr/local/mysql/bin/mysqld(+0xbb8f85)[0x55af7c603f85]
 
/usr/local/mysql/bin/mysqld(+0xad8bb8)[0x55af7c523bb8]
 
row/row0sel.cc:4777(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x55af7c3798f4]
 
handler/ha_innodb.cc:9279(ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function))[0x55af7c1b3cb7]
 
sql/handler.cc:2896(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function))[0x55af7c1a58b9]
 
sql/sql_select.cc:21124(join_read_always_key(st_join_table*))[0x55af7c1c8123]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x55af7c1c8353]
 
/usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55af7c1c65b6]
 
/usr/local/mysql/bin/mysqld(+0x6e5637)[0x55af7c130637]
 
/usr/local/mysql/bin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55af7c1301d4]
 
sql/sql_derived.cc:1200(mysql_derived_fill(THD*, LEX*, TABLE_LIST*))[0x55af7c1a5649]
 
/usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x178)[0x55af7c1a5888]
 
/usr/local/mysql/bin/mysqld(+0x74d07c)[0x55af7c19807c]
 
/usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1be)[0x55af7c1a58ce]
 
sql_select.cc:0(evaluate_join_record(JOIN*, st_join_table*, int))[0x55af7c1c8123]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x55af7c1c8353]
 
/usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55af7c1c65b6]
 
/usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1c7)[0x55af7c1c7117]
 
/usr/local/mysql/bin/mysqld(+0x6167d7)[0x55af7c0617d7]
 
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3a6a)[0x55af7c16d91a]
 
sql/sql_parse.cc:6360(execute_sqlcom_select(THD*, TABLE_LIST*))[0x55af7c172f7c]
 
/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1d9e)[0x55af7c1754ee]
 
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0x109)[0x55af7c176d79]
 
/usr/local/mysql/bin/mysqld(_Z11tp_callbackP13TP_connection+0x1d8)[0x55af7c347068]
 
/usr/local/mysql/bin/mysqld(+0xab4cc0)[0x55af7c4ffcc0]
 
sql/threadpool_common.cc:365(tp_callback(TP_connection*))[0x55af7c88fdcd]
 
pthread_create.c:0(start_thread)[0x7f2d9141cde5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f2d9027af1d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f2828075ad0): SELECT   ...

while executing complex enough SELECT involving the secondary index previously reported as problematic:

2020-07-16  9:26:30 165327 [ERROR] InnoDB: Clustered record for sec rec not found index `T_IDX02` of table `db`.`T`
 
InnoDB: sec index record PHYSICAL RECORD: n_fields 1; compact format; info bits 64
 
...
 
2020-07-16 09:26:30 0x7f70c44e6700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/rem/rem0rec.cc line 1982
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
200716  9:26:30 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.13-MariaDB-log
 
key_buffer_size=33554432
 
read_buffer_size=1048576
 
max_used_connections=235
 
max_threads=2010
 
thread_count=151
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10348902 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f6fd261a508
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f70c44e5de0 thread_stack 0x49000
 
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x55d0a511ee2e]
 
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x30f)[0x55d0a4b2fb3f]
 
/lib64/libpthread.so.0(+0xf5e0)[0x7f759cd8e5e0]
 
/lib64/libc.so.6(gsignal+0x37)[0x7f759bb1d277]
 
/lib64/libc.so.6(abort+0x148)[0x7f759bb1e968]
 
/usr/local/mysql/bin/mysqld(+0x629e48)[0x55d0a482fe48]
 
/usr/local/mysql/bin/mysqld(+0xb72683)[0x55d0a4d78683]
 
/usr/local/mysql/bin/mysqld(+0xc43ddf)[0x55d0a4e49ddf]
 
/usr/local/mysql/bin/mysqld(+0xbb9c00)[0x55d0a4dbfc00]
 
/usr/local/mysql/bin/mysqld(+0xad8bb8)[0x55d0a4cdebb8]
 
/usr/local/mysql/bin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x94)[0x55d0a4b348f4]
 
/usr/local/mysql/bin/mysqld(+0x768cb7)[0x55d0a496ecb7]
 
/usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1a9)[0x55d0a49608b9]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN10exec_innerEv+0xc73)[0x55d0a4983123]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x55d0a4983353]
 
/usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55d0a49815b6]
 
/usr/local/mysql/bin/mysqld(+0x6e5637)[0x55d0a48eb637]
 
/usr/local/mysql/bin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0x104)[0x55d0a48eb1d4]
 
/usr/local/mysql/bin/mysqld(_ZN13st_join_table12preread_initEv+0x79)[0x55d0a4960649]
 
/usr/local/mysql/bin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x178)[0x55d0a4960888]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN10exec_innerEv+0xc73)[0x55d0a4983123]
 
/usr/local/mysql/bin/mysqld(_ZN4JOIN4execEv+0x33)[0x55d0a4983353]
 
/usr/local/mysql/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55d0a49815b6]
 
/usr/local/mysql/bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1c7)[0x55d0a4982117]
 
/usr/local/mysql/bin/mysqld(+0x6167d7)[0x55d0a481c7d7]
 
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3a6a)[0x55d0a492891a]
 
/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x33c)[0x55d0a492df7c]
 
/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1d9e)[0x55d0a49304ee]
 
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0x109)[0x55d0a4931d79]
 
/usr/local/mysql/bin/mysqld(_Z11tp_callbackP13TP_connection+0x1d8)[0x55d0a4b02068]
 
/usr/local/mysql/bin/mysqld(+0xab4cc0)[0x55d0a4cbacc0]
 
/usr/local/mysql/bin/mysqld(+0xe44dcd)[0x55d0a504adcd]
 
/lib64/libpthread.so.0(+0x7de5)[0x7f759cd86de5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f759bbe4f1d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f70720c3200): SELECT ...

After dropping and recreating the index crashes stopped. The instance had never been restored from mariabackup. There are no virtual columns involved.

 Comments   
Comment by Marko Mäkelä [ 2020-07-16 ]

It is a large table that was recently initialized by loading an SQL dump, and there were no DDL operations on the table since then, only DML.

I have some suspicion that InnoDB change buffering could corrupt secondary indexes. Because the corrupted index is a non-unique secondary index, we cannot possibly blame SET unique_checks=0 for this.

I also have a suspicion that bugs in the adaptive hash index could corrupt indexes.

I do not know an exact mechanism for either corruption. Ideally, we should be lucky and repeat such corruption internally, with rr replay profile.

Was the server restarted between the table creation and the corruption?

Comment by MG [ 2020-07-23 ]

I can reproduce this in 10.3.23 and 10.4.13 with these settings:

[mysqld]
 
innodb_adaptive_hash_index=0
 
innodb_change_buffering=0


CREATE DATABASE /*!32312 IF NOT EXISTS*/ `selfinsert` /*!40100 DEFAULT CHARACTER SET latin1 */;
 
 
 
USE `selfinsert`;
 
DROP TABLE IF EXISTS `t1`;
 
SET SESSION innodb_strict_mode=1;
 
 
 
CREATE TABLE `t1` (
 
  `id` tinyint(3) unsigned NOT NULL AUTO_INCREMENT,
 
  `c2` tinyint(3) unsigned DEFAULT NULL,
 
  `c3` char(40) NOT NULL DEFAULT '',
 
  `c4` char(40) NOT NULL DEFAULT '',
 
  `x001` tinyint(3) unsigned DEFAULT NULL,
 
  `x002` tinyint(3) unsigned DEFAULT NULL,
 
  `x003` tinyint(3) unsigned DEFAULT NULL,
 
  `x004` tinyint(3) unsigned DEFAULT NULL,
 
  `x005` tinyint(3) unsigned DEFAULT NULL,
 
  `x006` tinyint(3) unsigned DEFAULT NULL,
 
  `x007` tinyint(3) unsigned DEFAULT NULL,
 
  `x008` tinyint(3) unsigned DEFAULT NULL,
 
  `x009` tinyint(3) unsigned DEFAULT NULL,
 
  `x010` tinyint(3) unsigned DEFAULT NULL,
 
  `x011` tinyint(3) unsigned DEFAULT NULL,
 
  `x012` tinyint(3) unsigned DEFAULT NULL,
 
  `x013` tinyint(3) unsigned DEFAULT NULL,
 
  `x014` tinyint(3) unsigned DEFAULT NULL,
 
  `x015` tinyint(3) unsigned DEFAULT NULL,
 
  `x016` tinyint(3) unsigned DEFAULT NULL,
 
  `x017` tinyint(3) unsigned DEFAULT NULL,
 
  `x018` tinyint(3) unsigned DEFAULT NULL,
 
  `x019` tinyint(3) unsigned DEFAULT NULL,
 
  `x020` tinyint(3) unsigned DEFAULT NULL,
 
  `x021` tinyint(3) unsigned DEFAULT NULL,
 
  `x022` tinyint(3) unsigned DEFAULT NULL,
 
  `x023` tinyint(3) unsigned DEFAULT NULL,
 
  `x024` tinyint(3) unsigned DEFAULT NULL,
 
  `x025` tinyint(3) unsigned DEFAULT NULL,
 
  `x026` tinyint(3) unsigned DEFAULT NULL,
 
  `x027` tinyint(3) unsigned DEFAULT NULL,
 
  `x028` tinyint(3) unsigned DEFAULT NULL,
 
  `x029` tinyint(3) unsigned DEFAULT NULL,
 
  `x030` tinyint(3) unsigned DEFAULT NULL,
 
  `x031` tinyint(3) unsigned DEFAULT NULL,
 
  `x032` tinyint(3) unsigned DEFAULT NULL,
 
  `x033` tinyint(3) unsigned DEFAULT NULL,
 
  `x034` tinyint(3) unsigned DEFAULT NULL,
 
  `x035` tinyint(3) unsigned DEFAULT NULL,
 
  `x036` tinyint(3) unsigned DEFAULT NULL,
 
  `x037` tinyint(3) unsigned DEFAULT NULL,
 
  `x038` tinyint(3) unsigned DEFAULT NULL,
 
  `x039` tinyint(3) unsigned DEFAULT NULL,
 
  `x040` tinyint(3) unsigned DEFAULT NULL,
 
  `x041` tinyint(3) unsigned DEFAULT NULL,
 
  `x042` tinyint(3) unsigned DEFAULT NULL,
 
  `x043` tinyint(3) unsigned DEFAULT NULL,
 
  `x044` tinyint(3) unsigned DEFAULT NULL,
 
  `x045` tinyint(3) unsigned DEFAULT NULL,
 
  `x046` tinyint(3) unsigned DEFAULT NULL,
 
  `x047` tinyint(3) unsigned DEFAULT NULL,
 
  `x048` tinyint(3) unsigned DEFAULT NULL,
 
  `x049` tinyint(3) unsigned DEFAULT NULL,
 
  `x050` tinyint(3) unsigned DEFAULT NULL,
 
  `x051` tinyint(3) unsigned DEFAULT NULL,
 
  `x052` tinyint(3) unsigned DEFAULT NULL,
 
  `x053` tinyint(3) unsigned DEFAULT NULL,
 
  `x054` tinyint(3) unsigned DEFAULT NULL,
 
  `x055` tinyint(3) unsigned DEFAULT NULL,
 
  `x056` tinyint(3) unsigned DEFAULT NULL,
 
  `x057` tinyint(3) unsigned DEFAULT NULL,
 
  `x058` tinyint(3) unsigned DEFAULT NULL,
 
  `x059` tinyint(3) unsigned DEFAULT NULL,
 
  `x060` tinyint(3) unsigned DEFAULT NULL,
 
  `x061` tinyint(3) unsigned DEFAULT NULL,
 
  `x062` tinyint(3) unsigned DEFAULT NULL,
 
  `x063` tinyint(3) unsigned DEFAULT NULL,
 
  `x064` tinyint(3) unsigned DEFAULT NULL,
 
  `x065` tinyint(3) unsigned DEFAULT NULL,
 
  `x066` tinyint(3) unsigned DEFAULT NULL,
 
  `x067` tinyint(3) unsigned DEFAULT NULL,
 
  `x068` tinyint(3) unsigned DEFAULT NULL,
 
  `x069` tinyint(3) unsigned DEFAULT NULL,
 
  `x070` tinyint(3) unsigned DEFAULT NULL,
 
  `x071` tinyint(3) unsigned DEFAULT NULL,
 
  `x072` tinyint(3) unsigned DEFAULT NULL,
 
  `x073` tinyint(3) unsigned DEFAULT NULL,
 
  `x074` tinyint(3) unsigned DEFAULT NULL,
 
  `x075` tinyint(3) unsigned DEFAULT NULL,
 
  `x076` tinyint(3) unsigned DEFAULT NULL,
 
  `x077` tinyint(3) unsigned DEFAULT NULL,
 
  `x078` tinyint(3) unsigned DEFAULT NULL,
 
  `x079` tinyint(3) unsigned DEFAULT NULL,
 
  `x080` tinyint(3) unsigned DEFAULT NULL,
 
  `x081` tinyint(3) unsigned DEFAULT NULL,
 
  `x082` tinyint(3) unsigned DEFAULT NULL,
 
  `x083` tinyint(3) unsigned DEFAULT NULL,
 
  `x084` tinyint(3) unsigned DEFAULT NULL,
 
  `x085` tinyint(3) unsigned DEFAULT NULL,
 
  `x086` tinyint(3) unsigned DEFAULT NULL,
 
  `x087` tinyint(3) unsigned DEFAULT NULL,
 
  `x088` tinyint(3) unsigned DEFAULT NULL,
 
  `x089` tinyint(3) unsigned DEFAULT NULL,
 
  `x090` tinyint(3) unsigned DEFAULT NULL,
 
  `x091` tinyint(3) unsigned DEFAULT NULL,
 
  `x092` tinyint(3) unsigned DEFAULT NULL,
 
  `x093` tinyint(3) unsigned DEFAULT NULL,
 
  `x094` tinyint(3) unsigned DEFAULT NULL,
 
  `x095` tinyint(3) unsigned DEFAULT NULL,
 
  `x096` tinyint(3) unsigned DEFAULT NULL,
 
  `x097` tinyint(3) unsigned DEFAULT NULL,
 
  `x098` tinyint(3) unsigned DEFAULT NULL,
 
  `x099` tinyint(3) unsigned DEFAULT NULL,
 
  `x100` tinyint(3) unsigned DEFAULT NULL,
 
  `x101` tinyint(3) unsigned DEFAULT NULL,
 
  `x102` tinyint(3) unsigned DEFAULT NULL,
 
  `x103` tinyint(3) unsigned DEFAULT NULL,
 
  `x104` tinyint(3) unsigned DEFAULT NULL,
 
  `x105` tinyint(3) unsigned DEFAULT NULL,
 
  `x106` tinyint(3) unsigned DEFAULT NULL,
 
  `x107` tinyint(3) unsigned DEFAULT NULL,
 
  `x108` tinyint(3) unsigned DEFAULT NULL,
 
  `x109` tinyint(3) unsigned DEFAULT NULL,
 
  `x110` tinyint(3) unsigned DEFAULT NULL,
 
  `x111` tinyint(3) unsigned DEFAULT NULL,
 
  `x112` tinyint(3) unsigned DEFAULT NULL,
 
  `x113` tinyint(3) unsigned DEFAULT NULL,
 
  `x114` tinyint(3) unsigned DEFAULT NULL,
 
  `x115` tinyint(3) unsigned DEFAULT NULL,
 
  `x116` tinyint(3) unsigned DEFAULT NULL,
 
  `x117` tinyint(3) unsigned DEFAULT NULL,
 
  `x118` tinyint(3) unsigned DEFAULT NULL,
 
  `x119` tinyint(3) unsigned DEFAULT NULL,
 
  `x120` tinyint(3) unsigned DEFAULT NULL,
 
  `x121` tinyint(3) unsigned DEFAULT NULL,
 
  `x122` tinyint(3) unsigned DEFAULT NULL,
 
  `x123` tinyint(3) unsigned DEFAULT NULL,
 
  `x124` tinyint(3) unsigned DEFAULT NULL,
 
  `x125` tinyint(3) unsigned DEFAULT NULL,
 
  `x126` tinyint(3) unsigned DEFAULT NULL,
 
  `x127` tinyint(3) unsigned DEFAULT NULL,
 
  `x128` tinyint(3) unsigned DEFAULT NULL,
 
  `x129` tinyint(3) unsigned DEFAULT NULL,
 
  `x130` tinyint(3) unsigned DEFAULT NULL,
 
  `x131` tinyint(3) unsigned DEFAULT NULL,
 
  `x132` tinyint(3) unsigned DEFAULT NULL,
 
  `x133` tinyint(3) unsigned DEFAULT NULL,
 
  `x134` tinyint(3) unsigned DEFAULT NULL,
 
  `x135` tinyint(3) unsigned DEFAULT NULL,
 
  `x136` tinyint(3) unsigned DEFAULT NULL,
 
  `x137` tinyint(3) unsigned DEFAULT NULL,
 
  `x138` tinyint(3) unsigned DEFAULT NULL,
 
  `x139` tinyint(3) unsigned DEFAULT NULL,
 
  `x140` tinyint(3) unsigned DEFAULT NULL,
 
  `x141` tinyint(3) unsigned DEFAULT NULL,
 
  `x142` tinyint(3) unsigned DEFAULT NULL,
 
  `x143` tinyint(3) unsigned DEFAULT NULL,
 
  `x144` tinyint(3) unsigned DEFAULT NULL,
 
  `x145` tinyint(3) unsigned DEFAULT NULL,
 
  `x146` tinyint(3) unsigned DEFAULT NULL,
 
  `x147` tinyint(3) unsigned DEFAULT NULL,
 
  `x148` tinyint(3) unsigned DEFAULT NULL,
 
  `x149` tinyint(3) unsigned DEFAULT NULL,
 
  `x150` tinyint(3) unsigned DEFAULT NULL,
 
  `x151` tinyint(3) unsigned DEFAULT NULL,
 
  `x152` tinyint(3) unsigned DEFAULT NULL,
 
  `x153` tinyint(3) unsigned DEFAULT NULL,
 
  `x154` tinyint(3) unsigned DEFAULT NULL,
 
  `x155` tinyint(3) unsigned DEFAULT NULL,
 
  `x156` tinyint(3) unsigned DEFAULT NULL,
 
  `x157` tinyint(3) unsigned DEFAULT NULL,
 
  `x158` tinyint(3) unsigned DEFAULT NULL,
 
  `x159` tinyint(3) unsigned DEFAULT NULL,
 
  `x160` tinyint(3) unsigned DEFAULT NULL,
 
  `x161` tinyint(3) unsigned DEFAULT NULL,
 
  `x162` tinyint(3) unsigned DEFAULT NULL,
 
  `x163` tinyint(3) unsigned DEFAULT NULL,
 
  `x164` tinyint(3) unsigned DEFAULT NULL,
 
  `x165` tinyint(3) unsigned DEFAULT NULL,
 
  `x166` tinyint(3) unsigned DEFAULT NULL,
 
  `x167` tinyint(3) unsigned DEFAULT NULL,
 
  `x168` tinyint(3) unsigned DEFAULT NULL,
 
  `x169` tinyint(3) unsigned DEFAULT NULL,
 
  `x170` tinyint(3) unsigned DEFAULT NULL,
 
  `x171` tinyint(3) unsigned DEFAULT NULL,
 
  `x172` tinyint(3) unsigned DEFAULT NULL,
 
  `x173` tinyint(3) unsigned DEFAULT NULL,
 
  `x174` tinyint(3) unsigned DEFAULT NULL,
 
  `x175` tinyint(3) unsigned DEFAULT NULL,
 
  `x176` tinyint(3) unsigned DEFAULT NULL,
 
  `x177` tinyint(3) unsigned DEFAULT NULL,
 
  `x178` tinyint(3) unsigned DEFAULT NULL,
 
  `x179` tinyint(3) unsigned DEFAULT NULL,
 
  `x180` tinyint(3) unsigned DEFAULT NULL,
 
  `x181` tinyint(3) unsigned DEFAULT NULL,
 
  `x182` tinyint(3) unsigned DEFAULT NULL,
 
  `x183` tinyint(3) unsigned DEFAULT NULL,
 
  `x184` tinyint(3) unsigned DEFAULT NULL,
 
  `x185` tinyint(3) unsigned DEFAULT NULL,
 
  `x186` tinyint(3) unsigned DEFAULT NULL,
 
  `x187` tinyint(3) unsigned DEFAULT NULL,
 
  `x188` tinyint(3) unsigned DEFAULT NULL,
 
  `x189` tinyint(3) unsigned DEFAULT NULL,
 
  `x190` tinyint(3) unsigned DEFAULT NULL,
 
  `x191` tinyint(3) unsigned DEFAULT NULL,
 
  `x192` tinyint(3) unsigned DEFAULT NULL,
 
  `x193` tinyint(3) unsigned DEFAULT NULL,
 
  `x194` tinyint(3) unsigned DEFAULT NULL,
 
  `x195` tinyint(3) unsigned DEFAULT NULL,
 
  `x196` tinyint(3) unsigned DEFAULT NULL,
 
  `x197` tinyint(3) unsigned DEFAULT NULL,
 
  `x198` tinyint(3) unsigned DEFAULT NULL,
 
  `x199` tinyint(3) unsigned DEFAULT NULL,
 
  `x200` tinyint(3) unsigned DEFAULT NULL,
 
  `x201` tinyint(3) unsigned DEFAULT NULL,
 
  `x202` tinyint(3) unsigned DEFAULT NULL,
 
  `x203` tinyint(3) unsigned DEFAULT NULL,
 
  `x204` tinyint(3) unsigned DEFAULT NULL,
 
  `x205` tinyint(3) unsigned DEFAULT NULL,
 
  `x206` tinyint(3) unsigned DEFAULT NULL,
 
  `x207` tinyint(3) unsigned DEFAULT NULL,
 
  `x208` tinyint(3) unsigned DEFAULT NULL,
 
  `x209` tinyint(3) unsigned DEFAULT NULL,
 
  `x210` tinyint(3) unsigned DEFAULT NULL,
 
  `x211` tinyint(3) unsigned DEFAULT NULL,
 
  `x212` tinyint(3) unsigned DEFAULT NULL,
 
  `x213` tinyint(3) unsigned DEFAULT NULL,
 
  `x214` tinyint(3) unsigned DEFAULT NULL,
 
  `x215` tinyint(3) unsigned DEFAULT NULL,
 
  `x216` tinyint(3) unsigned DEFAULT NULL,
 
  `x217` tinyint(3) unsigned DEFAULT NULL,
 
  `x218` tinyint(3) unsigned DEFAULT NULL,
 
  `x219` tinyint(3) unsigned DEFAULT NULL,
 
  `x220` tinyint(3) unsigned DEFAULT NULL,
 
  `x221` tinyint(3) unsigned DEFAULT NULL,
 
  `x222` tinyint(3) unsigned DEFAULT NULL,
 
  `x223` tinyint(3) unsigned DEFAULT NULL,
 
  `x224` tinyint(3) unsigned DEFAULT NULL,
 
  `x225` tinyint(3) unsigned DEFAULT NULL,
 
  `x226` tinyint(3) unsigned DEFAULT NULL,
 
  `x227` tinyint(3) unsigned DEFAULT NULL,
 
  `x228` tinyint(3) unsigned DEFAULT NULL,
 
  `x229` tinyint(3) unsigned DEFAULT NULL,
 
  `x230` tinyint(3) unsigned DEFAULT NULL,
 
  `x231` tinyint(3) unsigned DEFAULT NULL,
 
  `x232` tinyint(3) unsigned DEFAULT NULL,
 
  `x233` tinyint(3) unsigned DEFAULT NULL,
 
  `x234` tinyint(3) unsigned DEFAULT NULL,
 
  `x235` tinyint(3) unsigned DEFAULT NULL,
 
  `x236` tinyint(3) unsigned DEFAULT NULL,
 
  `x237` tinyint(3) unsigned DEFAULT NULL,
 
  `x238` tinyint(3) unsigned DEFAULT NULL,
 
  `x239` tinyint(3) unsigned DEFAULT NULL,
 
  `x240` tinyint(3) unsigned DEFAULT NULL,
 
  `x241` tinyint(3) unsigned DEFAULT NULL,
 
  `x242` tinyint(3) unsigned DEFAULT NULL,
 
  `x243` tinyint(3) unsigned DEFAULT NULL,
 
  `x244` tinyint(3) unsigned DEFAULT NULL,
 
  `x245` tinyint(3) unsigned DEFAULT NULL,
 
  `x246` tinyint(3) unsigned DEFAULT NULL,
 
  `x247` tinyint(3) unsigned DEFAULT NULL,
 
  `x248` tinyint(3) unsigned DEFAULT NULL,
 
  `x249` tinyint(3) unsigned DEFAULT NULL,
 
  `x250` tinyint(3) unsigned DEFAULT NULL,
 
  `x251` tinyint(3) unsigned DEFAULT NULL,
 
  `x252` tinyint(3) unsigned DEFAULT NULL,
 
  `x253` tinyint(3) unsigned DEFAULT NULL,
 
  `x254` tinyint(3) unsigned DEFAULT NULL,
 
  `x255` tinyint(3) unsigned DEFAULT NULL,
 
  `x256` tinyint(3) unsigned DEFAULT NULL,
 
  `x257` tinyint(3) unsigned DEFAULT NULL,
 
  `x258` tinyint(3) unsigned DEFAULT NULL,
 
  `x259` tinyint(3) unsigned DEFAULT NULL,
 
  `x260` tinyint(3) unsigned DEFAULT NULL,
 
  `x261` tinyint(3) unsigned DEFAULT NULL,
 
  `x262` tinyint(3) unsigned DEFAULT NULL,
 
  `x263` tinyint(3) unsigned DEFAULT NULL,
 
  `x264` tinyint(3) unsigned DEFAULT NULL,
 
  `x265` tinyint(3) unsigned DEFAULT NULL,
 
  `x266` tinyint(3) unsigned DEFAULT NULL,
 
  `x267` tinyint(3) unsigned DEFAULT NULL,
 
  `x268` tinyint(3) unsigned DEFAULT NULL,
 
  `x269` tinyint(3) unsigned DEFAULT NULL,
 
  `x270` tinyint(3) unsigned DEFAULT NULL,
 
  `x271` tinyint(3) unsigned DEFAULT NULL,
 
  `x272` tinyint(3) unsigned DEFAULT NULL,
 
  `x273` tinyint(3) unsigned DEFAULT NULL,
 
  `x274` tinyint(3) unsigned DEFAULT NULL,
 
  `x275` tinyint(3) unsigned DEFAULT NULL,
 
  `x276` tinyint(3) unsigned DEFAULT NULL,
 
  `x277` tinyint(3) unsigned DEFAULT NULL,
 
  `x278` tinyint(3) unsigned DEFAULT NULL,
 
  `x279` tinyint(3) unsigned DEFAULT NULL,
 
  `x280` tinyint(3) unsigned DEFAULT NULL,
 
  `x281` tinyint(3) unsigned DEFAULT NULL,
 
  `x282` tinyint(3) unsigned DEFAULT NULL,
 
  `x283` tinyint(3) unsigned DEFAULT NULL,
 
  `x284` tinyint(3) unsigned DEFAULT NULL,
 
  `x285` tinyint(3) unsigned DEFAULT NULL,
 
  `x286` tinyint(3) unsigned DEFAULT NULL,
 
  `x287` tinyint(3) unsigned DEFAULT NULL,
 
  `x288` tinyint(3) unsigned DEFAULT NULL,
 
  `x289` tinyint(3) unsigned DEFAULT NULL,
 
  `x290` tinyint(3) unsigned DEFAULT NULL,
 
  `x291` tinyint(3) unsigned DEFAULT NULL,
 
  `x292` tinyint(3) unsigned DEFAULT NULL,
 
  PRIMARY KEY (`id`),
 
  KEY `ix_c3` (`c3`)
 
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
 
 
 
begin;
 
insert into `t1` (c2) values ('123');
 
update t1 set c3 = sha1(concat(c2));
 
update t1 set c4 = sha1(concat(c2));
 
select id, c2, c3, c4 from t1 force index(primary)\G
 
insert into t1 (c3) select c3 from t1 force index(ix_c3);

Workarounds include:

  • Do a COMMIT
  • IGNORE INDEX(ix_c3)
  • DROP any x column
  • Do not modify c4 within the transaction (crash seems intermittent if you do not modify c4)
Comment by MG [ 2020-07-23 ]

Replaying (parsed) general log or using the previously provided `selfinsert` there is no crash in 10.3.18 or 10.3.21. Both 10.3.22 and 10.3.23 crash in either scenario (production sql or contrived case)

Comment by MG [ 2020-07-29 ]

Here is the crash on 10.3.23:

2020-07-29  0:37:35 0 [Note] /usr/sbin/mysqld: ready for connections.
 
Version: '10.3.23-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
 
200729  0:38:09 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.23-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=7
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467423 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f16c0000f38
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f16c45a4d50 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5640fdc8401e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x5640fd71d01f]
 
sigaction.c:0(__restore_rt)[0x7f16d2d4c7e0]
 
:0(__memmove_avx_unaligned_erms)[0x7f16d10b0480]
 
/usr/sbin/mysqld(+0x9db1c2)[0x5640fd9581c2]
 
/usr/sbin/mysqld(+0xa6bc01)[0x5640fd9e8c01]
 
/usr/sbin/mysqld(+0xa39b47)[0x5640fd9b6b47]
 
/usr/sbin/mysqld(+0x985210)[0x5640fd902210]
 
/usr/sbin/mysqld(+0x9879fc)[0x5640fd9049fc]
 
/usr/sbin/mysqld(+0x4e1bfc)[0x5640fd45ebfc]
 
/usr/sbin/mysqld(+0xa22182)[0x5640fd99f182]
 
/usr/sbin/mysqld(+0x948633)[0x5640fd8c5633]
 
/usr/sbin/mysqld(+0x92e7b7)[0x5640fd8ab7b7]
 
/usr/sbin/mysqld(_ZN7handler14ha_index_firstEPh+0x19f)[0x5640fd72326f]
 
/usr/sbin/mysqld(+0x605aff)[0x5640fd582aff]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x5640fd577c49]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa8b)[0x5640fd59a93b]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5640fd59ab53]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x5640fd59905a]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x5640fd599b6c]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x84a8)[0x5640fd547588]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1fb)[0x5640fd547e8b]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xf76)[0x5640fd549716]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11b)[0x5640fd54b37b]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1d6)[0x5640fd61fd36]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x5640fd61fe4d]
 
pthread_create.c:0(start_thread)[0x7f16d2d4240b]
 
:0(__GI___clone)[0x7f16d1051e7f]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f16c000f940): is an invalid pointer
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway.
 
Query: insert into t1 (c3) select c3 from t1 force index(ix_c3)

And after upgrading 10.4.23 on the same box, and creating a new datadir:

2020-07-29  0:42:08 9 [ERROR] [FATAL] InnoDB: Unable to allocate memory of size 4294961976.
 
200729  0:42:08 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.23-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=7
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467423 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f2800000f38
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f28105c2d50 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559b366d101e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x559b3616a01f]
 
sigaction.c:0(__restore_rt)[0x7f281ed6a7e0]
 
:0(__GI_raise)[0x7f281cfb6ae0]
 
:0(__GI_abort)[0x7f281cfb7f88]
 
/usr/sbin/mysqld(+0xa8fcf3)[0x559b36459cf3]
 
/usr/sbin/mysqld(+0x9a3ac0)[0x559b3636dac0]
 
/usr/sbin/mysqld(+0x9a3b2c)[0x559b3636db2c]
 
/usr/sbin/mysqld(+0xa6bf44)[0x559b36435f44]
 
/usr/sbin/mysqld(+0xa39b47)[0x559b36403b47]
 
/usr/sbin/mysqld(+0x985210)[0x559b3634f210]
 
/usr/sbin/mysqld(+0x9879fc)[0x559b363519fc]
 
/usr/sbin/mysqld(+0x4e1bfc)[0x559b35eabbfc]
 
/usr/sbin/mysqld(+0xa22182)[0x559b363ec182]
 
/usr/sbin/mysqld(+0x948633)[0x559b36312633]
 
/usr/sbin/mysqld(+0x92e7b7)[0x559b362f87b7]
 
/usr/sbin/mysqld(_ZN7handler14ha_index_firstEPh+0x19f)[0x559b3617026f]
 
/usr/sbin/mysqld(+0x605aff)[0x559b35fcfaff]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x559b35fc4c49]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa8b)[0x559b35fe793b]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x559b35fe7b53]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x559b35fe605a]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x559b35fe6b6c]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x84a8)[0x559b35f94588]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1fb)[0x559b35f94e8b]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xf76)[0x559b35f96716]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11b)[0x559b35f9837b]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1d6)[0x559b3606cd36]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x559b3606ce4d]
 
pthread_create.c:0(start_thread)[0x7f281ed6040b]
 
:0(__GI___clone)[0x7f281d06fe7f]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f280000f940): is an invalid pointer
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway.
 
Query: insert into t1 (c3) select c3 from t1 force index(ix_c3)

Note that neither of the above mention row_search_mvcc() - this is odd because on a different server (cent6/2.6.32 instead of amzn2/4.14.173), the same SQL file does get this in the stack trace:

2020-07-29  0:44:23 10 [ERROR] [FATAL] InnoDB: Unable to allocate memory of size 4294869168.
 
200729  0:44:23 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.13-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=7
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467752 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7fbeff0bc008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fbf03fa1d10 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x555d675f394e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x555d6708aa3f]
 
/lib64/libpthread.so.0(+0x335600f7e0)[0x7fbf039f07e0]
 
/lib64/libc.so.6(gsignal+0x35)[0x7fbf0204a4f5]
 
/lib64/libc.so.6(abort+0x175)[0x7fbf0204bcd5]
 
/usr/sbin/mysqld(+0xc016c0)[0x555d6737d6c0]
 
:0(ib_vector_create(ib_alloc_t*, unsigned long, unsigned long))[0x555d6729a650]
 
mem/mem0mem.cc:320(mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long))[0x555d6729a6bc]
 
mem/mem0mem.cc:381(mem_heap_add_block(mem_block_info_t*, unsigned long))[0x555d6735ab2f]
 
include/mem0mem.ic:193(mem_heap_alloc)[0x555d6732a8b7]
 
row/row0vers.cc:196(row_vers_impl_x_locked_low)[0x555d6727b250]
 
lock/lock0lock.cc:1233(lock_sec_rec_some_has_impl)[0x555d6727fadc]
 
lock/lock0lock.cc:5757(lock_sec_rec_read_check_and_lock(unsigned long, buf_block_t const*, unsigned char const*, dict_index_t*, unsigned short const*, lock_mode, unsigned long, que_thr_t*))[0x555d66d898f2]
 
row/row0sel.cc:1279(sel_set_rec_lock(btr_pcur_t*, unsigned char const*, dict_index_t*, unsigned short const*, unsigned long, unsigned long, que_thr_t*, mtr_t*))[0x555d6731a009]
 
row/row0sel.cc:5065(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x555d6723a7f7]
 
handler/ha_innodb.cc:9279(index_read)[0x555d6709020f]
 
sql/handler.cc:2987(handler::ha_index_first(unsigned char*))[0x555d66ec68df]
 
sql/sql_select.cc:21372(join_read_first(st_join_table*))[0x555d66ebb779]
 
sql/sql_select.cc:20364(sub_select(JOIN*, st_join_table*, bool))[0x555d66eddfe3]
 
sql/sql_select.cc:19905(do_select)[0x555d66ede213]
 
sql/sql_select.cc:4242(JOIN::exec())[0x555d66edc476]
 
sql/sql_select.cc:4675(mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x555d66edcfd7]
 
sql/sql_select.cc:422(handle_select(THD*, LEX*, select_result*, unsigned long))[0x555d66e875d7]
 
sql/sql_parse.cc:4642(mysql_execute_command(THD*))[0x555d66e88e3c]
 
sql/sql_parse.cc:7900(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x555d66e8b3ae]
 
sql/sql_audit.h:169(mysql_audit_general)[0x555d66e8cc39]
 
sql/sql_parse.cc:1360(do_command(THD*))[0x555d66f6bb4a]
 
sql/sql_connect.cc:1412(do_handle_one_connection(CONNECT*))[0x555d66f6bc2d]
 
/lib64/libpthread.so.0(+0x3356007aa1)[0x7fbf039e8aa1]
 
/lib64/libc.so.6(clone+0x6d)[0x7fbf02100c4d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fbeff0f6020): is an invalid pointer
 
Connection ID (thread ID): 10
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
 
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway.
 
Query: insert into t1 (c3) select c3 from t1 force index(ix_c3)

Anyway, I think what I have posted is related to the original bug report but I suppose it could be different issues. If I am not mistaken, my original testing did have a crash a bit more similar to the original crash. In other words, this SQL is seeming to crash in different ways depending on the environment but does always crash instantly.

Comment by MG [ 2020-07-29 ]

My apologies, here is that amzn2 box upgraded to 10.4 (previously yum wanted "manual" upgrade steps):

2020-07-29  0:56:24 9 [ERROR] [FATAL] InnoDB: Unable to allocate memory of size 4294910304.
 
200729  0:56:24 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.13-MariaDB
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=1
 
max_threads=153
 
thread_count=7
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467752 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7fd518000f38
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fd51c0bad10 thread_stack 0x49000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55b0cb0656ce]
 
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55b0caafbf7f]
 
sigaction.c:0(__restore_rt)[0x7fd52a1e37e0]
 
:0(__GI_raise)[0x7fd52842fae0]
 
:0(__GI_abort)[0x7fd528430f88]
 
/usr/sbin/mysqld(+0xba2160)[0x55b0cadee160]
 
/usr/sbin/mysqld(+0xabec90)[0x55b0cad0ac90]
 
/usr/sbin/mysqld(+0xabecfc)[0x55b0cad0acfc]
 
/usr/sbin/mysqld(+0xb7f5af)[0x55b0cadcb5af]
 
/usr/sbin/mysqld(+0xb4f277)[0x55b0cad9b277]
 
/usr/sbin/mysqld(+0xa9f140)[0x55b0caceb140]
 
/usr/sbin/mysqld(+0xaa39cc)[0x55b0cacef9cc]
 
/usr/sbin/mysqld(+0x5b2683)[0x55b0ca7fe683]
 
/usr/sbin/mysqld(+0xb3e9c9)[0x55b0cad8a9c9]
 
/usr/sbin/mysqld(+0xa5e6a3)[0x55b0cacaa6a3]
 
/usr/sbin/mysqld(+0xa5e8b3)[0x55b0cacaa8b3]
 
/usr/sbin/mysqld(_ZN7handler14ha_index_firstEPh+0x19f)[0x55b0cab0174f]
 
/usr/sbin/mysqld(+0x6eedff)[0x55b0ca93adff]
 
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1a9)[0x55b0ca92fd39]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc73)[0x55b0ca952223]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55b0ca952453]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55b0ca9506a6]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x55b0ca951217]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x819f)[0x55b0ca8fcebf]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1fb)[0x55b0ca8fd8ab]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1957)[0x55b0ca8ffa37]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x109)[0x55b0ca901049]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x55b0ca9debfa]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55b0ca9decdd]
 
pthread_create.c:0(start_thread)[0x7fd52a1d940b]
 
:0(__GI___clone)[0x7fd5284e8e7f]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fd518010c70): insert into t1 (c3) select c3 from t1 force index(ix_c3)
 
Connection ID (thread ID): 9
 
Status: NOT_KILLED

Still no mention of row_search_mvcc() for the crash while this SQL mentions this is stack trace on another Linux kernel (2.6.32-754.25.1.el6.x86_64 + 10.4.13-MariaDB)

Comment by Marko Mäkelä [ 2020-08-03 ]

mg, I think that your comments must duplicate MDEV-23198, which has already been fixed. (Sorry, I did not notice your test case until now.)

For the original reported failure in this issue, which is related to this code in row_search_mvcc(), I am afraid that more information would be needed.

        /*-------------------------------------------------------------*/
 
        /* Do sanity checks in case our cursor has bumped into page
 
        corruption */
 
 
 
        if (comp) {
 
                if (rec_get_info_bits(rec, true) & REC_INFO_MIN_REC_FLAG) {

I have no idea how rec would not be safe to dereference here. Could the record list be corrupted in such a way that it is not terminated by the 'supremum' pseudo-record but instead by a null pointer?

This message that valerii posted does indicate secondary index corruption:

mariadb-10.4.13

 
2020-07-16  9:26:30 165327 [ERROR] InnoDB: Clustered record for sec rec not found index `T_IDX02` of table `db`.`T`
 
 
 
InnoDB: sec index record PHYSICAL RECORD: n_fields 1; compact format; info bits 64

The most promising recent development that is related to this is in MDEV-22924. I analyzed a ‘fake trace’ that was basically a duplicate of the MDEV-15532 scenario (explicit use of XA START, with wrong COMMIT). But, mleich claims that he had repeated the problem without involving any XA command. I hope that we will get a new rr replay trace in a few weeks.

In the above output, the info bits 64 looks suspicious too. Normally, the info bits in a secondary index leaf page record can only be 0 or 32. (On the leftmost non-leaf pages in the tree, the first record must have info bits 16, and in any non-leaf record, the bits 32 (delete-mark flag) are garbage and may be set or unset. But this should be a leaf page record.) The bit value 64 is not assigned.

Comment by Marko Mäkelä [ 2020-09-07 ]

MDEV-22924 fixes a bug in an optimization of MVCC reads in secondary indexes (MDEV-20301), and I do not think that it can explain this corruption.

Comment by Elena Stepanova [ 2020-09-14 ]

See a test case in the comment to MDEV-23723 .

The stack trace there matches the very first stack trace in the description here, so it may be related. It doesn't however cause the corruption errors and other problems described here.

Comment by Marko Mäkelä [ 2020-11-30 ]

The simple CTE test case in MDEV-23723 looks like something else. I do not think that any InnoDB corruption should be involved in that case.

Comment by Marko Mäkelä [ 2021-03-09 ]

valerii, I think that any secondary index page can be corrupted due to MDEV-24449. I think that that bug (which was only recently fixed, and is present in all earlier InnoDB releases) can also cause arbitrary pages in the system tablespace to be corrupted.

Does this still occur after upgrading to 10.4.18 or 10.5?

Comment by Alice Sherepa [ 2021-04-19 ]

looks similar to MDEV-25442 (MDEV-23723), the query there is causing crash or corruption, like:

2021-04-19 10:57:22 5 [ERROR] InnoDB: Clustered record for sec rec not found index `idx_auction_lots_05` of table `test`.`auction_lots_test`
 
InnoDB: sec index record PHYSICAL RECORD: n_fields 5; compact format; info bits 0
 
 0: len 8; hex 8000000000018c73; asc        s;;
 
 1: len 30; hex 8000000000dca0c400000000000080000000000000800000000000005f52; asc                             _R; (total 255 bytes);
 
 2: len 8; hex 800000000000005f; asc        _;;
 
 3: len 30; hex 52808000000000000000800000000000000080000000000000004e415f64; asc R                         NA_d; (total 239 bytes);
 
 4: len 0; hex ; asc ;;

Comment by Marko Mäkelä [ 2023-02-09 ]

One of many possible causes of this could be MDEV-30009 and the InnoDB change buffer.

Generated at Thu Feb 08 09:20:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.