[MDEV-24489] Server crashes: double free or corruption Created: 2020-12-24  Updated: 2023-05-12  Resolved: 2023-05-12

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.27
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: RINAT MAKHMUTOV Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 2
Labels: crash
Environment:

Linux 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 20072 5968 454 129 13649 13580
Swap: 20477 182 20295


Attachments: Text File error.09.01.2021.log     Text File error.12.01.2021.log     Text File error.14.12.log     Text File error.15.12.log     Text File error.17.12.log     Text File error.log    
Issue Links:
Relates
relates to MDEV-22956 rpl.rpl_old_master failed in buildbot... Open
relates to MDEV-24505 innodb.innodb-ucs2 test failed, asse... Closed
relates to MDEV-24495 InnoDB: Assertion failure in file ut0... Closed
relates to MDEV-25394 MariaDB crashing every few days Closed
relates to MDEV-25963 MariaDB crashed without apparent reason Open

 Description   

I couldn't reproduce it manually, but it crashes 1-3 times per day, after that 5 days stable, and again. With same settings

double free or corruption (!prev)
2020-12-23 21:30:01 0x7fd010c1d700  InnoDB: Assertion failure in file /build/mariadb-10.3-Zg5BW3/mariadb-10.3-10.3.27/storage/innobase/include/ut0lst.h line 334
InnoDB: Failing assertion: list.count > 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.

double free or corruption (!prev)
201217  6:40:01 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.27-MariaDB-0+deb10u1-log
key_buffer_size=1317011456
read_buffer_size=131072
max_used_connections=50
max_threads=802
thread_count=47
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3049343 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f11140c89f8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f11d01c6dd8 thread_stack 0x30000

Config:

[mysqld]
sql-mode=""
skip-networking=0
skip-bind-address
max_connections        = 800
max_connect_errors = 1000000
skip-external-locking
 
character-set-server=utf8
collation-server=utf8_general_ci
 
slow_query_log=ON
slow_query_log_file=/var/log/mysql/mysql-slow.log
long_query_time=1
 
# old config:
key_buffer		= 1256M
max_allowed_packet	= 320M
thread_stack		= 192K
thread_cache_size       = 8
myisam-recover         = BACKUP
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit	= 320M
query_cache_size        = 640M
expire_logs_days	= 10
max_binlog_size         = 100M



 Comments   
Comment by Marko Mäkelä [ 2023-04-14 ]

Do you have any reproducible test case for this?

I would recommend building the server using cmake -DWITH_ASAN=ON, so that this type of errors could be caught earlier. It is possible that the memory has been corrupted by some write outside of InnoDB. The bulk of open heap-use-after-free or heap-use-after-poison bugs are outside InnoDB.

Generated at Thu Feb 08 09:30:22 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.