Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.6.5
-
None
-
AWS (db.r5.4xlarge, 16 cpu, 128GB), linux,
Description
Can't provide much information as to why this happens: neither application server nor db server have had recent changes. We did notice a disk space warning, but not sure we actually ran out of disk space.
All of a sudden the MariaDB server crashed and restarted. It now seems to crash every couple of hours, rather unpredictable, seems to occur more on higher load.
From the crash report:
Error:
|
2022-06-03 14:05:50 0x144c13cb3700 InnoDB: Assertion failure in file /local/p4clients/pkgbuild-WHiDx/workspace/src/RDSMariaDB/storage/innobase/rem/rem0rec.cc line 877
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
220603 14:05:50 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.5-MariaDB-log
|
key_buffer_size=16777216
|
read_buffer_size=262144
|
max_used_connections=332
|
max_threads=16002
|
thread_count=333
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 37277516 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x144c0a26bed8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x144c13cb1598 thread_stack 0x40000
|
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x556e070d03fe]
|
/rdsdbbin/mysql/bin/mysqld(handle_fatal_signal+0x51b)[0x556e068ca88b]
|
/lib64/libpthread.so.0(+0x117e0)[0x146406c1a7e0]
|
/lib64/libc.so.6(gsignal+0x110)[0x146406891c20]
|
:0(__GI_raise)[0x1464068930c8]
|
/rdsdbbin/mysql/bin/mysqld(+0x74bf8a)[0x556e065ccf8a]
|
ut/ut0rbt.cc:460(rbt_eject_node(ib_rbt_node_t*, ib_rbt_node_t*) [clone .part.14])[0x556e06eac6d5]
|
rem/rem0rec.cc:682(rec_init_offsets)[0x556e06e817e9]
|
page/page0cur.cc:1357(page_cur_insert_rec_low(page_cur_t const*, dict_index_t*, unsigned char const*, unsigned short*, mtr_t*))[0x556e06f68955]
|
btr/btr0cur.cc:3554(btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned short**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, unsigned long, que_thr_t*, mtr_t*))[0x556e06ec36b1]
|
row/row0ins.cc:3109(row_ins_sec_index_entry_low(unsigned long, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*))[0x556e06ec945b]
|
row/row0ins.cc:3313(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool))[0x556e06ec98e4]
|
row/row0ins.cc:3672(row_ins)[0x556e06edb0be]
|
row/row0mysql.cc:1322(row_insert_for_mysql(unsigned char const*, row_prebuilt_t*, ins_mode_t))[0x556e06e1cecb]
|
handler/ha_innodb.cc:7841(ha_innobase::write_row(unsigned char const*))[0x556e068d6144]
|
sql/handler.cc:7524(handler::ha_write_row(unsigned char const*))[0x556e06691341]
|
sql/sql_insert.cc:2146(write_record(THD*, TABLE*, st_copy_info*, select_result*))[0x556e0669af87]
|
sql/sql_insert.cc:1124(mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool, select_result*))[0x556e066c43b7]
|
sql/sql_parse.cc:4601(mysql_execute_command(THD*, bool))[0x556e066c8159]
|
sql/sql_parse.cc:8154(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x556e066bf2fa]
|
sql/sql_parse.cc:1970(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x556e066be437]
|
sql/sql_parse.cc:1414(do_command(THD*, bool))[0x556e067b5f83]
|
sql/sql_connect.cc:1461(do_handle_one_connection(CONNECT*, bool))[0x556e067b62a4]
|
sql/sql_connect.cc:1359(handle_one_connection)[0x556e06d9031c]
|
/lib64/libpthread.so.0(+0x740b)[0x146406c1040b]
|
/lib64/libc.so.6(clone+0x3f)[0x14640694b09f]
|
|
I did a CHECK TABLE on the table that seems to cause the crashes, this resulted in dozens of
[ERROR] Got error 180 when reading table './db@002dquery/the_table_name'
|