[MDEV-28748] Server crash on INSERT INTO, resulted in 180 when reading table. Created: 2022-06-05  Updated: 2022-07-20  Resolved: 2022-07-20

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.6.5
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Bart Vansegbroeck Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Environment:

AWS (db.r5.4xlarge, 16 cpu, 128GB), linux,



 Description   

Can't provide much information as to why this happens: neither application server nor db server have had recent changes. We did notice a disk space warning, but not sure we actually ran out of disk space.

All of a sudden the MariaDB server crashed and restarted. It now seems to crash every couple of hours, rather unpredictable, seems to occur more on higher load.

From the crash report:

Error:
2022-06-03 14:05:50 0x144c13cb3700 InnoDB: Assertion failure in file /local/p4clients/pkgbuild-WHiDx/workspace/src/RDSMariaDB/storage/innobase/rem/rem0rec.cc line 877
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220603 14:05:50 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.6.5-MariaDB-log
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=332
max_threads=16002
thread_count=333
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 37277516 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x144c0a26bed8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x144c13cb1598 thread_stack 0x40000
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x556e070d03fe]
/rdsdbbin/mysql/bin/mysqld(handle_fatal_signal+0x51b)[0x556e068ca88b]
/lib64/libpthread.so.0(+0x117e0)[0x146406c1a7e0]
/lib64/libc.so.6(gsignal+0x110)[0x146406891c20]
:0(__GI_raise)[0x1464068930c8]
/rdsdbbin/mysql/bin/mysqld(+0x74bf8a)[0x556e065ccf8a]
ut/ut0rbt.cc:460(rbt_eject_node(ib_rbt_node_t*, ib_rbt_node_t*) [clone .part.14])[0x556e06eac6d5]
rem/rem0rec.cc:682(rec_init_offsets)[0x556e06e817e9]
page/page0cur.cc:1357(page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*))[0x556e06f68955]
btr/btr0cur.cc:3554(btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*))[0x556e06ec36b1]
row/row0ins.cc:3109(row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*))[0x556e06ec945b]
row/row0ins.cc:3313(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool))[0x556e06ec98e4]
row/row0ins.cc:3672(row_ins)[0x556e06edb0be]
row/row0mysql.cc:1322(row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t))[0x556e06e1cecb]
handler/ha_innodb.cc:7841(ha_innobase::write_row(unsigned char const*))[0x556e068d6144]
sql/handler.cc:7524(handler::ha_write_row(unsigned char const*))[0x556e06691341]
sql/sql_insert.cc:2146(write_record(THD*, TABLE*, st_copy_info*, select_result*))[0x556e0669af87]
sql/sql_insert.cc:1124(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*))[0x556e066c43b7]
sql/sql_parse.cc:4601(mysql_execute_command(THD*, bool))[0x556e066c8159]
sql/sql_parse.cc:8154(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x556e066bf2fa]
sql/sql_parse.cc:1970(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x556e066be437]
sql/sql_parse.cc:1414(do_command(THD*, bool))[0x556e067b5f83]
sql/sql_connect.cc:1461(do_handle_one_connection(CONNECT*, bool))[0x556e067b62a4]
sql/sql_connect.cc:1359(handle_one_connection)[0x556e06d9031c]
/lib64/libpthread.so.0(+0x740b)[0x146406c1040b]
/lib64/libc.so.6(clone+0x3f)[0x14640694b09f]

I did a CHECK TABLE on the table that seems to cause the crashes, this resulted in dozens of

 [ERROR] Got error 180 when reading table './db@002dquery/the_table_name'



 Comments   
Comment by Daniel Black [ 2022-06-06 ]

perror 180

MariaDB error code 180: Index corrupted

Can you include a SHOW CREATE TABLE for that table to indicate what datatypes and indexes are involved?

Comment by Bart Vansegbroeck [ 2022-06-07 ]

Below is the anonimized version of the table.

CHECK TABLE als yielded the following:
[ERROR] InnoDB: Flagged corruption of `idx_E_H` in table `db_name`.`the_table_name` in CHECK TABLE;

CREATE TABLE `the_table_name` (
  `A` char(36) CHARACTER SET ascii NOT NULL,
  `B` char(3) CHARACTER SET ascii DEFAULT NULL,
  `C` decimal(8,2) DEFAULT NULL,
  `D` varchar(255) CHARACTER SET utf8mb4 DEFAULT NULL,
  `E` char(36) DEFAULT NULL,
  `F` varchar(255) CHARACTER SET ascii DEFAULT NULL,
  `G` varchar(255) CHARACTER SET ascii DEFAULT NULL,
  `H` datetime DEFAULT NULL,
  `I` text DEFAULT NULL,
  `J` text DEFAULT NULL,
  `K` text DEFAULT NULL,
  `L` mediumtext DEFAULT NULL,
  `M` varchar(20) NOT NULL DEFAULT 'SSS',
  `N` varchar(255) DEFAULT NULL,
  `O` varchar(255) CHARACTER SET ascii DEFAULT NULL,
  `P` varchar(36) DEFAULT NULL,
  `Q` varchar(36) DEFAULT NULL,
  `R` int(11) DEFAULT NULL,
  `S` varchar(255) DEFAULT NULL,
  `T` varchar(255) DEFAULT NULL,
  `U` varchar(255) DEFAULT NULL,
  `V` varchar(255) DEFAULT NULL,
  `W` varchar(255) DEFAULT NULL,
  `X` varchar(255) DEFAULT NULL,
  `Y` varchar(255) DEFAULT NULL,
  `Z` varchar(40) DEFAULT NULL,
  `AA` varchar(36) DEFAULT NULL,
  `BB` enum('BB1','BB2','BB3','BB4','BB5') DEFAULT NULL,
  `CC` text DEFAULT NULL,
  `DD` text CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `EE` tinyint(1) DEFAULT NULL,
  PRIMARY KEY (`A`),
  KEY `idx_E_H` (`E`,`H`),
  KEY `idx_Q_P_F_H` (`Q`,`P`,`F`,`H`),
  KEY `idx_AA` (`AA`),
  KEY `idx_E_P` (`E`,`P`),
  KEY `idx_Q_R` (`Q`,`R`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3;

Comment by Alice Sherepa [ 2022-06-07 ]

Is it possible for you to upgrade to the recent version of MariaDB and see if this helps? (10.6.8 )
(MDEV-25283/MDEV-27993)

Comment by Bart Vansegbroeck [ 2022-06-07 ]

Yes, we will roll out the update, will keep you posted here.

Generated at Thu Feb 08 10:03:10 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.