[MDEV-23670] Crash during OPTIMIZE TABLE mysql.innodb_table_stats Created: 2020-09-04  Updated: 2021-11-23  Resolved: 2021-09-17

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Alter Table, Storage Engine - InnoDB
Affects Version/s: 10.0, 10.1.34, 10.3.23, 10.4.17, 10.2, 10.5
Fix Version/s: 10.6.5

Type: Bug Priority: Critical
Reporter: Joris de Leeuw Assignee: Marko Mäkelä
Resolution: Won't Fix Votes: 1
Labels: crash, optimizer, race
Environment:

OS: CloudLinux like RHEL - Kernel: 3.10.0-962.3.2.lve1.5.32.el6h.x86_64

Attachments: Text File mysql crashlog 2020-09-04 08-10-17.txt    
Issue Links:
Relates
relates to MDEV-19539 Assertion failure in file /home/build... Closed
relates to MDEV-15020 Server hangs due to InnoDB persistent... Closed
relates to MDEV-24579 Error table->get_ref_count() after up... Closed
relates to MDEV-25702 InnoDB: Failing assertion: rec_offs_n... Closed
relates to MDEV-25919 InnoDB reports misleading lock wait t... Closed

 Description   

We run rather large production servers with over hundreds of databases with varying sizes between a few MB and many GB.

To improve performance we optimize tables periodically with a cron running this command:

/usr/bin/mysqlcheck --optimize --all-databases --auto-repair

We started noticing problems with crashing MariaDB servers during the optimize using MariaDB 10.1. We already switched to MariaDB 10.3 in the meantime.

The problem keeps returning almost every optimize with MariaDB 10.3 on our most busy production servers. On less busy servers the problem seems more rare. In a test environments we are unable to reproduce the issue.

Log:

2020-09-04 08:10:17 0x7fc16c0b0700  InnoDB: Assertion failure in file /builddir/build/BUILD/mariadb-10.3.23/storage/innobase/row/row0merge.cc line 4492
 
InnoDB: Failing assertion: table->get_ref_count() == 0
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
200904  8:10:17 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.3.23-MariaDB-log-cll-lve
 
key_buffer_size=67108864
 
read_buffer_size=1048576
 
max_used_connections=253
 
max_threads=502
 
thread_count=49
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1104963 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7fc0b0178be8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fc16c0afd68 thread_stack 0x40000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55f8152d3d5e]
 
*** stack smashing detected ***: /usr/sbin/mysqld terminated

Config:

mysqld would have been started with the following arguments:
 
--basedir=/usr --bind-address=:: --binlog_checksum=NONE --binlog_format=STATEMENT --datadir=/var/lib/mysql --expire_logs_days=10 --ft_min_word_len=3 --innodb_buffer_pool_size=256M --innodb_checksum_algorithm=innodb --innodb_doublewrite=0 --innodb_file_format=barracuda --innodb_file_per_table=1 --innodb_large_prefix=ON --innodb_log_file_size=192M --innodb_strict_mode=false --innodb_use_native_aio=0 --join_buffer_size=1M --key_buffer_size=64M --local-infile=1 --log-error=/var/log/mysqld.log --log_warnings=2 --long_query_time=2 --max_allowed_packet=24M --max_binlog_size=100M --max_connections=500 --max_heap_table_size=20M --max_user_connections=100 --myisam_sort_buffer_size=32M --open_files_limit=51200 --pid-file=/var/run/mysqld/mysqld.pid --port=3306 --query_cache_size=32M --read_buffer_size=1M --read_rnd_buffer_size=1M --skip-external-locking --slow_query_log=1 --slow_query_log_file=/var/lib/mysql/slow_query.log --socket=/var/lib/mysql_sock/mysql.sock --sort_buffer_size=1M --sql_mode=NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION --ssl --ssl-cert=/etc/pki/tls/certs/server.fullchain --ssl-key=/etc/pki/tls/private/server.mysql.key --ssl_cipher=TLSv1.2 --symbolic-links=0 --table_cache=2048 --table_definition_cache=2048 --thread_cache_size=8 --thread_stack=256K --tmp_table_size=10M --tmpdir=/var/lib/mysql_tmp --user=mysql



 Comments   
Comment by Joris de Leeuw [ 2020-12-18 ]

We still see the same behaviour with MariaDB 10.4.17. We hoped upgrading resolved this issue.

It doesn't however happens every time that MariaDB crashes during an optimize, but seem to still happen more often on our more busy production servers. On our test environments we don't see this behaviour.

2020-12-16 08:47:45 0x7fe400249700  InnoDB: Assertion failure in file /builddir/build/BUILD/mariadb-10.4.17/storage/innobase/row/row0merge.cc line 4487
 
InnoDB: Failing assertion: table->get_ref_count() == 0
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
201216  8:47:45 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
 
 
Server version: 10.4.17-MariaDB-log-cll-lve
 
key_buffer_size=67108864
 
read_buffer_size=1048576
 
max_used_connections=400
 
max_threads=502
 
thread_count=112
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1106055 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
 
 
Thread pointer: 0x7fe320ffdb18
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fe400248d38 thread_stack 0x40000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55660c83783e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x55660c32eecd]
 
sigaction.c:0(__restore_rt)[0x7fe4791237e0]
 
/lib64/libc.so.6(gsignal+0x35)[0x7fe47755c4f5]
 
/lib64/libc.so.6(abort+0x175)[0x7fe47755dcd5]
 
/usr/sbin/mysqld(+0x5b5f7c)[0x55660c026f7c]
 
/usr/sbin/mysqld(+0x5a528f)[0x55660c01628f]
 
/usr/sbin/mysqld(+0xb8581e)[0x55660c5f681e]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPK25st_mysql_const_lex_stringS3_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2f2b)[0x55660c1c208b]
 
/usr/sbin/mysqld(_Z20mysql_recreate_tableP3THDP10TABLE_LISTb+0x1bd)[0x55660c1c364d]
 
/usr/sbin/mysqld(+0x7abc0d)[0x55660c21cc0d]
 
/usr/sbin/mysqld(+0x7ad49d)[0x55660c21e49d]
 
/usr/sbin/mysqld(_ZN22Sql_cmd_optimize_table7executeEP3THD+0xa0)[0x55660c21f4f0]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1539)[0x55660c126109]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x295)[0x55660c12d1f5]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1561)[0x55660c12f6e1]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x104)[0x55660c130c94]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x25e)[0x55660c21222e]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x55660c2122fd]
 
/lib64/libpthread.so.0(+0x7aa1)[0x7fe47911baa1]
 
/lib64/libc.so.6(clone+0x6d)[0x7fe477612c2d]
 
 
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fe320795240): OPTIMIZE TABLE `innodb_table_stats`
 
 
 
 
 
Connection ID (thread ID): 12958239
 
Status: NOT_KILLED
 
 
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
 
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /var/lib/mysql
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             514477               514477               processes 
Max open files            51200                51200                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       514477               514477               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: /tmp/core-%e-sig%s-user%u-group%g-pid%p-time%t

Comment by Joris de Leeuw [ 2021-02-19 ]

We still see sporadically MySQL crashes during a long running optimize on our production servers.
What is interesting that in this case it seemed to fail at OPTIMIZE TABLE `innodb_index_stats`

2021-02-17 15:00:54 0x7f9efc6f4700 InnoDB: Assertion failure in file /builddir/build/BUILD/mariadb-10.4.17/storage/innobase/row/row0merge.cc line 4487
 
InnoDB: Failing assertion: table->get_ref_count() == 0
 
InnoDB: We intentionally generate a memory trap.
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
InnoDB: If you get repeated assertion failures or crashes, even
 
InnoDB: immediately after the mysqld startup, there may be
 
InnoDB: corruption in the InnoDB tablespace. Please refer to
 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
InnoDB: about forcing recovery.
 
210217 15:00:54 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.4.17-MariaDB-log-cll-lve
 
key_buffer_size=67108864
 
read_buffer_size=1048576
 
max_used_connections=501
 
max_threads=502
 
thread_count=275
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1106055 K bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f9c70d6b478
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f9efc6f3d38 thread_stack 0x40000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x559fcc19b83e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x559fcbc92ecd]
 
/lib64/libpthread.so.0(+0xf7e0)[0x7f9f3dfa57e0]
 
/lib64/libc.so.6(gsignal+0x35)[0x7f9f3c3de4f5]
 
/lib64/libc.so.6(abort+0x175)[0x7f9f3c3dfcd5]
 
/usr/sbin/mysqld(+0x5b5f7c)[0x559fcb98af7c]
 
/usr/sbin/mysqld(+0x5a528f)[0x559fcb97a28f]
 
/usr/sbin/mysqld(+0xb8581e)[0x559fcbf5a81e]
 
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPK25st_mysql_const_lex_stringS3_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2f2b)[0x559fcbb2608b]
 
/usr/sbin/mysqld(_Z20mysql_recreate_tableP3THDP10TABLE_LISTb+0x1bd)[0x559fcbb2764d]
 
/usr/sbin/mysqld(+0x7abc0d)[0x559fcbb80c0d]
 
/usr/sbin/mysqld(+0x7ad49d)[0x559fcbb8249d]
 
/usr/sbin/mysqld(_ZN22Sql_cmd_optimize_table7executeEP3THD+0xa0)[0x559fcbb834f0]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1539)[0x559fcba8a109]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x295)[0x559fcba911f5]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1561)[0x559fcba936e1]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x104)[0x559fcba94c94]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x25e)[0x559fcbb7622e]
 
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x559fcbb762fd]
 
/lib64/libpthread.so.0(+0x7aa1)[0x7f9f3df9daa1]
 
/lib64/libc.so.6(clone+0x6d)[0x7f9f3c494c2d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f9c70b95040): OPTIMIZE TABLE `innodb_index_stats`
 
 
 
Connection ID (thread ID): 9798309
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /var/lib/mysql
 
Resource Limits:
 
Limit Soft Limit Hard Limit Units
 
Max cpu time unlimited unlimited seconds
 
Max file size unlimited unlimited bytes
 
Max data size unlimited unlimited bytes
 
Max stack size 8388608 unlimited bytes
 
Max core file size 0 unlimited bytes
 
Max resident set unlimited unlimited bytes
 
Max processes 513221 513221 processes
 
Max open files 51200 51200 files
 
Max locked memory 65536 65536 bytes
 
Max address space unlimited unlimited bytes
 
Max file locks unlimited unlimited locks
 
Max pending signals 513221 513221 signals
 
Max msgqueue size 819200 819200 bytes
 
Max nice priority 0 0
 
Max realtime priority 0 0
 
Max realtime timeout unlimited unlimited us
 
Core pattern: core

Comment by Marko Mäkelä [ 2021-02-19 ]

Does this only happen on the tables mysql.innodb_table_stats or mysql.innodb_index_stats? Those are a little special, and executing DDL (such as OPTIMIZE) on them is risky. MDEV-15020 should fix this by protecting the InnoDB internal access to the statistics tables with metadata lock (MDL).

If the problem occurs or other tables, then I would be interested to know more. For the statistics tables, this is not really a surprise to me.

Comment by Joris de Leeuw [ 2021-02-26 ]

All crashes in the last few months on our production servers always seem to happen due to optimize on the 'innodb_index_stats' table and in some cases on the 'innodb_table_stats' table.

We will exclude them in our optimize to see if this mitigates the problem.

Comment by Elena Stepanova [ 2021-03-28 ]

Joriz,

Has excluding system tables from optimization process helped?

Comment by Joris de Leeuw [ 2021-03-28 ]

Yes. excluding the mysql databases in the optimize helps. We didn't see any crashes any more.

We now use the following command:

/usr/bin/mysqlcheck --optimize --all-databases --auto-repair --skip-database=mysql

So it is wise to skip the mysql database by default.

Comment by Valerii Kravchuk [ 2021-06-01 ]

We have yet another case of a similar stack trace and crash on OPTIMIZE (normal table, not anyone inside mysql database):

210509  7:26:12 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.1.34-MariaDB
 
key_buffer_size=33554432
 
read_buffer_size=262144
 
max_used_connections=27
 
max_threads=2002
 
thread_count=13
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1611965 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7ec35f431008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7eff9cfb2080 thread_stack 0x48400
 
*** buffer overflow detected ***: /data/mysql/bin/mysqld terminated
 
======= Backtrace: =========
 
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f00348fb577]
 
/lib64/libc.so.6(+0x1166f2)[0x7f00348f96f2]
 
/lib64/libc.so.6(+0x1184d7)[0x7f00348fb4d7]
 
/data/mysql/bin/mysqld(my_addr_resolve+0xd0)[0x5606dd9c6ee0]
 
/data/mysql/bin/mysqld(my_print_stacktrace+0x1c2)[0x5606dd9b3152]
 
/data/mysql/bin/mysqld(handle_fatal_signal+0x305)[0x5606dd4d6875]
 
/lib64/libpthread.so.0(+0xf630)[0x7f003677c630]
 
/data/mysql/bin/mysqld(+0x8a1095)[0x5606dd7b5095]
 
/data/mysql/bin/mysqld(+0x8b9784)[0x5606dd7cd784]
 
/data/mysql/bin/mysqld(+0x8b99d5)[0x5606dd7cd9d5]
 
/data/mysql/bin/mysqld(+0x8bad78)[0x5606dd7ced78]
 
/data/mysql/bin/mysqld(+0x8bf639)[0x5606dd7d3639]
 
/data/mysql/bin/mysqld(+0x81c77d)[0x5606dd73077d]
 
/data/mysql/bin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x24d9)[0x5606dd3dd839]
 
/data/mysql/bin/mysqld(_Z20mysql_recreate_tableP3THDP10TABLE_LISTb+0x15d)[0x5606dd3df0cd]
 
/data/mysql/bin/mysqld(+0x51af13)[0x5606dd42ef13]
 
/data/mysql/bin/mysqld(_ZN22Sql_cmd_optimize_table7executeEP3THD+0xec)[0x5606dd42ff7c]
 
/data/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1315)[0x5606dd349705]
 
/data/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x342)[0x5606dd351a72]
 
/data/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x250a)[0x5606dd35510a]
 
/data/mysql/bin/mysqld(_Z10do_commandP3THD+0x177)[0x5606dd355a97]
 
/data/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x5606dd422a1a]
 
/data/mysql/bin/mysqld(handle_one_connection+0x40)[0x5606dd422bc0]
 
/data/mysql/bin/mysqld(+0x7517dd)[0x5606dd6657dd]
 
/lib64/libpthread.so.0(+0x7ea5)[0x7f0036774ea5]
 
/lib64/libc.so.6(clone+0x6d)[0x7f00348e18dd]

Comment by Marko Mäkelä [ 2021-07-23 ]

I would need a stack trace of all threads during the crash, to analyze this. Please try to preserve the core dump for further questions (I may need to run some debugger commands to extract more information).

Comment by Marko Mäkelä [ 2021-09-14 ]

Joriz, I believe that running any DDL on the InnoDB statistics tables always was crash-prone before the fix of MDEV-25919 in the upcoming MariaDB Server 10.6.5 release. OPTIMIZE TABLE is internally implemented as ALTER TABLE…FORCE.

valerii, does the table of the support customer contain FULLTEXT INDEX? If yes, that could be a duplicate of MDEV-25702. Do we have fully resolved stack traces of those crashes?

Comment by Marko Mäkelä [ 2021-09-17 ]

Joriz, I believe that the originally reported issue (crash during OPTIMIZE TABLE mysql.innodb_index_stats or OPTIMIZE TABLE mysql.innodb_table_stats) has been fixed in MDEV-25919 in the upcoming 10.6.5 release. The fix is that any internal InnoDB operation that would access the statistics tables will acquire a shared metadata lock (MDL) on the statistics table names. That will prevent concurrent execution of such background operations and DDL (such as OPTIMIZE TABLE or ALTER TABLE) on the statistics tables.

In older release series this bug is not feasible to fix, because that fix depends on many large refactoring tasks that were only implemented in the 10.6 release series.

The crash that a support customer reported for 10.1.34 may or may not have occurred due to the same reason. There is no such assertion expression table->get_ref_count() == 0 in the 10.1 source code.

Comment by Jan Ingvoldstad [ 2021-11-12 ]

This issue also appears to apply to MariaDB 10.5.13. After rebooting a Debian system with 10.5.13, crashed tables are automatically attempted repaired, and this results in crashes a few times per minute with the following error:

InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.5.13/storage/innobase/row/row0merge.cc line 4338

After upgrading to MariaDB 10.6.5, it appears stable, repairing tables does not crash the server.

It would therefore be nice if 10.5.13 was added to the list of affected versions where this crasher will not be fixed.

Comment by Marko Mäkelä [ 2021-11-23 ]

I do not think that it is feasible to port the fix (MDEV-25919) from 10.6 to earlier versions. Any releases between 10.0 and 10.5 are affected by this.

Generated at Thu Feb 08 09:24:09 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.