[MDEV-19539] Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.15/storage/innobase/row/row0merge.cc line 4509 Created: 2019-05-21  Updated: 2021-09-17  Resolved: 2021-09-17

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Alter Table, Storage Engine - InnoDB
Affects Version/s: 10.3.15
Fix Version/s: 10.6.5

Type: Bug Priority: Major
Reporter: David L Assignee: Marko Mäkelä
Resolution: Won't Fix Votes: 0
Labels: None
Environment:

CentOS Linux release 7.6.1810 (Core)
3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
###########################################
Second crash on host003
OpenVz Container
CentOS Linux release 7.6.1810 (Core)
2.6.32-042stab134.8 #1 SMP Fri Dec 7 17:16:09 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux


Attachments: File mariadb-host003.log     File mariadb-host003.var     File mariadb.log.bad     File var.mariadb    
Issue Links:
Relates
relates to MDEV-15776 Failing assertion: table->get_ref_cou... Closed
relates to MDEV-23670 Crash during OPTIMIZE TABLE mysql.inn... Closed
relates to MDEV-24579 Error table->get_ref_count() after up... Closed

 Description   

Hello!

After upgrade mariadb 5.5 to mariadb 10.3.15.
I ran mysqlcheck -A -o and after checking 2 hours mariadb crashed with next log.

2019-05-21 15:25:21 0x7f0dc9ab5700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.15/storage/innobase/row/row0merge.cc line 4509
InnoDB: Failing assertion: table->get_ref_count() == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
190521 15:25:21 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.15-MariaDB
key_buffer_size=134217728
read_buffer_size=4194304
max_used_connections=147
max_threads=202
thread_count=150
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1169834 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x56033a84edf8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0dc9ab4d30 thread_stack 0x49000
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f0e86d9e9e7]
/lib64/libc.so.6(+0x115b62)[0x7f0e86d9cb62]
/lib64/libc.so.6(+0x117947)[0x7f0e86d9e947]
/usr/sbin/mysqld(my_addr_resolve+0xda)[0x56032e82d47a]
/usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x56032e816ab2]
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56032e2b353f]
/lib64/libpthread.so.0(+0xf5d0)[0x7f0e889e95d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f0e86cbd207]
/lib64/libc.so.6(abort+0x148)[0x7f0e86cbe8f8]
/usr/sbin/mysqld(+0x4c5e9c)[0x56032dffae9c]
/usr/sbin/mysqld(+0x9d9d7f)[0x56032e50ed7f]
/usr/sbin/mysqld(+0x939f47)[0x56032e46ef47]
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPK25st_mysql_const_lex_stringS3_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2ab0)[0x56032e16b180]
/usr/sbin/mysqld(_Z20mysql_recreate_tableP3THDP10TABLE_LISTb+0x191)[0x56032e16c821]
/usr/sbin/mysqld(+0x68e6c1)[0x56032e1c36c1]
/usr/sbin/mysqld(_ZN22Sql_cmd_optimize_table7executeEP3THD+0xdb)[0x56032e1c470b]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x3a04)[0x56032e0de404]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x56032e0e3bdb]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1d3e)[0x56032e0e68de]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x56032e0e773e]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1aa)[0x56032e1b879a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x56032e1b88bd]
/lib64/libpthread.so.0(+0x7dd5)[0x7f0e889e1dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f0e86d84ead]
 
Today After upgrade mariadb 5.5 to mariadb 10.3.15 in openvz container.
I ran mysqlcheck -A -o  and  after checking 10 seconds mariadb crashed with log (mariadb-host003.log)
 
And on another host is the same. 
mysql.column_stats                                 Table is already up to date
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.gtid_slave_pos
note     : Table does not support optimize, doing recreate + analyze instead
status   : OK
mysql.help_category                                Table is already up to date
mysql.help_keyword                                 Table is already up to date
mysql.help_relation                                Table is already up to date
mysql.help_topic                                   Table is already up to date
mysql.host                                         OK
mysql.index_stats                                  Table is already up to date
mysqlcheck: Got error: 2013: Lost connection to MySQL server during query when executing 'OPTIMIZE TABLE ... '



 Comments   
Comment by Marko Mäkelä [ 2021-09-17 ]

This looks exactly the same as MDEV-23670. Based on the output, I assume that the server was executing one of the following statements during the crash:

OPTIMIZE TABLE mysql.innodb_index_stats;
OPTIMIZE TABLE mysql.innodb_table_stats;

The current table name is not shown, but some other system tables were previously optimized.

I believe that this has been fixed by MDEV-25919 in the 10.6.5 release. Unfortunately, it is not feasible to port that fix to earlier release series.

Generated at Thu Feb 08 08:52:28 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.