[MDEV-26388] assertion (innodb) fseg_inode_try_get returns null in fseg_inode_get Created: 2021-08-17  Updated: 2021-09-20  Resolved: 2021-09-20

Status: Closed
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.5.12
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Loic Quillot Assignee: Daniel Black
Resolution: Duplicate Votes: 1
Labels: crash
Environment:

FreeBSD 12.2 + base ZFS + jail


Issue Links:
Duplicate
duplicates MDEV-26537 InnoDB corrupts files due to incorrec... Closed

 Description   

Dear Sirs,
After upgrading 3 of our mariadb servers from 10.5.11 to 10.5.12 we encounter this bug after around a week from the upgrade.
The mariadb servers are running in a standard default configuration and upgraded regularly for several years.
We thought that the bug may be related to this modification : MDEV-24393, so we disabled the skip_external_locking trying to have the advisory lock back but in vain, we still have the bug, mainly with the INSERT and CREATE statements.

Here is a log of one of the cases :

InnoDB: Assertion failure in file /usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/fsp/fsp0fsp.cc line 1535   
InnoDB: Failing assertion: inode                                                                                                                                                              
InnoDB: We intentionally generate a memory trap.                                                                                                                                              
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/                                                                                                                             InnoDB: If you get repeated assertion failures or crashes, even                                                                                                                               
InnoDB: immediately after the mysqld startup, there may be                                                                                                                                    
InnoDB: corruption in the InnoDB tablespace. Please refer to                                                                                                                                  
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/                                                                                                                              
InnoDB: about forcing recovery.                                                                                                                                                               
210816 13:40:17 [ERROR] mysqld got signal 6 ;                                                                                                                                                 
This could be because you hit a bug. It is also possible that this binary                      
or one of the libraries it was linked against is corrupt, improperly built,                                                                                                                   
or misconfigured. This error can also be caused by malfunctioning hardware.                                                                                                                   
                                                                                                                                                                                              
To report this bug, see https://mariadb.com/kb/en/reporting-bugs           
                                                                                                                                                                                              
We will try our best to scrape up some info that will hopefully help                        
diagnose the problem, but since we have already crashed,                                       
something is definitely wrong and this may fail.                                               
                                                                                               
Server version: 10.5.12-MariaDB                                                                
key_buffer_size=1073741824                                                                     
read_buffer_size=2097152                                                                                                                                                                      
max_used_connections=56                                                                                                                                                                       
max_threads=1026                                                                                                                                                                              
thread_count=61                                                                                                                                                                               
It is possible that mysqld could use up to                                                                                                                                                    
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5276648 K  bytes of memory                         
Hope that's ok; if not, decrease some variables in the equation.                                                                                                                              
                                                                                                                                                                                              
Thread pointer: 0x8edaba698                                                                                                                                                                   
Attempting backtrace. You can use the following information to find out                                                                                                                       
where mysqld died. If you see no messages after this, something went                                                                                                                          
terribly wrong...                                                                                                                                                                             
stack_bottom = 0x7fffdb35ae50 thread_stack 0x49000                                                                                                                                            
0x132cdfc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd                                                                                                                           
0xc75cbe <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd                                                                                                                           
0x80191bd20 <_pthread_sigmask+0x530> at /lib/libthr.so.3 
Trying to get some variables.                                                                  
Some pointers may be invalid and cause the dump to abort.                                                                                                                                     
Query (0x0): (null)                                                                            
Connection ID (thread ID): 0                                                                   
Status: NOT_KILLED                                                                             
                                                                                               
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: 
 
Core pattern: %N.core

So, now we downgraded them back to 10.5.11, and working on a staging server to try to debug it.

Thank you for any help you can give us.

Regards
LQ



 Comments   
Comment by Daniel Black [ 2021-08-18 ]

Hi LQ,

Can you give examples of your CREATE/INSERT statements that are causing this?

If FreeBSD is generating core files can you use a debugger to get a full backtrace?

Comment by Loic Quillot [ 2021-08-18 ]

Hello Daniel,

Thank you for your reply, in fact when I searched the logs I found all types of statements creating the bug (select, check table ... etc).

Here is an example with a signal 10 in a select of a nextcloud instance :

 
mysqld got signal 10 ;                                                                                                                                                
This could be because you hit a bug. It is also possible that this binary                                                                                                                     
or one of the libraries it was linked against is corrupt, improperly built,                                                                                                                   
or misconfigured. This error can also be caused by malfunctioning hardware.                                                                                                                   
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.12-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2
max_threads=153
thread_count=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K  bytes of memory 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x822192d98
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdd742f38 thread_stack 0x49000
0x1337bac <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc830be <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x801926d22 <_pthread_sigmask+0x532> at /lib/libthr.so.3
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x8221d3ff0): SELECT `s`.*, `f`.`fileid`, `f`.`path`, `f`.`permissions` as `f_permissions`, `f`.`storage`, `f`.`path_hash`, `f`.`parent` as `f_parent`, `f`.`name`, `f`.`mimetype`, `f`
.`mimepart`, `f`.`size`, `f`.`mtime`, `f`.`storage_mtime`, `f`.`encrypted`, `f`.`unencrypted_size`, `f`.`etag`, `f`.`checksum`, `st`.`id` AS `storage_string_id` FROM `oc_share` `s` LEFT JOIN
 `oc_filecache` `f` ON `s`.`file_source` = `f`.`fileid` LEFT JOIN `oc_storages` `st` ON `f`.`storage` = `st`.`numeric_id` WHERE (`s`.`share_type` = '10') AND (`s`.`share_with` IN ('kp3m2zmg'
, 'fupnj39r')) AND ((`s`.`item_type` = 'file') OR (`s`.`item_type` = 'folder')) ORDER BY `s`.`id` ASC
 
Connection ID (thread ID): 6
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off                                    
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core

And another example with signal 6 :

 
InnoDB: Assertion failure in file /usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/btr/btr0pcur.cc line 520
InnoDB: Failing assertion: page_is_comp(next_page) == page_is_comp(page)                     
InnoDB: We intentionally generate a memory trap.                         
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/                                                                                                                             
InnoDB: If you get repeated assertion failures or crashes, even             
InnoDB: immediately after the mysqld startup, there may be                                                                                                                                    
InnoDB: corruption in the InnoDB tablespace. Please refer to                                 
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/                          
InnoDB: about forcing recovery.                                                                
210810 12:10:16 [ERROR] mysqld got signal 6 ;                                                  
This could be because you hit a bug. It is also possible that this binary                                                                                                                     
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.                                                                                                                   
                                                                                               
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
                                                                                               
We will try our best to scrape up some info that will hopefully help     
diagnose the problem, but since we have already crashed,           
something is definitely wrong and this may fail.                                
                                                                                               
Server version: 10.5.12-MariaDB                                                                
key_buffer_size=134217728                     
read_buffer_size=131072
max_used_connections=5
max_threads=153
thread_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467792 K  bytes of memory 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x822434418
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffddcaff38 thread_stack 0x49000
0x132cdfc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc75cbe <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x80191bd22 <_pthread_sigmask+0x532> at /lib/libthr.so.3
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x8224b3a70): DELETE FROM `oc_share` WHERE `item_type` in ('file', 'folder') AND NOT EXISTS (SELECT `fileid` FROM `oc_filecache` WHERE `file_source` = `fileid`)
 
Connection ID (thread ID): 172
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off                                    
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core

And here an example with an insert :

mysqld got signal 10 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
                                               
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
                                               
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
                                               
Server version: 10.5.12-MariaDB  
key_buffer_size=1073741824
read_buffer_size=2097152
max_used_connections=81
max_threads=1026
thread_count=86
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5276648 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x8f99b5018
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffdbe7ff38 thread_stack 0x49000
0x132cdfc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc75cbe <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x80191bd20 <_pthread_sigmask+0x530> at /lib/libthr.so.3
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x8f99c78e0): insert into `sg` (`id`, `aid`, `title`, `score`, `rat`, `uat`, `cat`) values (?, ?, ?, ?, ?, ?, ?)
 
Connection ID (thread ID): 346
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core

and an example with update :

[ERROR] mysqld got signal 6 ;                                                                                                                                                 
This could be because you hit a bug. It is also possible that this binary                                                                                                                     
or one of the libraries it was linked against is corrupt, improperly built,                                                                                                                   
or misconfigured. This error can also be caused by malfunctioning hardware.                                                                                                                   
                                                                                                                                                                                              
To report this bug, see https://mariadb.com/kb/en/reporting-bugs                                                                                                                              
                                                                                                                                                                                              
We will try our best to scrape up some info that will hopefully help                                                                                                                          
diagnose the problem, but since we have already crashed,                                                                                                                                      
something is definitely wrong and this may fail.                                                                                                                                              
                                                                                                                                                                                              
Server version: 10.5.12-MariaDB                                                                                                                                                               
key_buffer_size=1073741824                                                                                                                                                                    
read_buffer_size=2097152                                                                                                                                                                      
max_used_connections=15                                                                                                                                                                       max_threads=1026                                                                               
thread_count=20                                                                                                                                                                               It is possible that mysqld could use up to                                                                                                                                                    
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5276648 K  bytes of memory                                                                                              Hope that's ok; if not, decrease some variables in the equation.                                                                                                                              
                                                                                                                                                                                              
Thread pointer: 0x8f7e78d58                                                                                                                                                                   
Attempting backtrace. You can use the following information to find out                                                                                                                       
where mysqld died. If you see no messages after this, something went                                                                                                                          
terribly wrong...                                                                                                                                                                             
stack_bottom = 0x7fffdd493f38 thread_stack 0x49000                                                                                                                                            
0x132cdfc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd                                                                                                                           
0xc75cbe <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x80191bd20 <_pthread_sigmask+0x530> at /lib/libthr.so.3
                                                                                               
Trying to get some variables.                                                                  
Some pointers may be invalid and cause the dump to abort.
Query (0x8f7ebd9b0): UPDATE `smon` SET `sstatus` = '2', `sresult` = 'a:5:{i:1;s:46:\"xxxxxxxxxxxxxxx\";i:6;s:61:\"xxxxxxxxxxxxxxxxxxxxxx\";i:4;s:53:\"xxxxxxxxxxxxxxx\";i:2;s:40:\"xxxxxxxxxxxxxxxxxxx\";i:3;s:51:\"xxxxxxxxxxxxxxxxxxxx\";}' WHERE (sid = 315)                                        
                                                                                               
Connection ID (thread ID): 29254                                                               
Status: NOT_KILLED                                                                                                                                                                            
                                                                                                                                                                                              
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off                                                                                                                                   
                                                                                               
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core

I have the core dump, here is the backtrace of a signal 6 :

# gdb /usr/local/libexec/mariadbd mariadbd.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
Core was generated by `/usr/local/libexec/mariadbd --defaults-extra-file=/usr/local/etc/mysql/my.cnf --'.
Program terminated with signal 6, Aborted.
#0  0x0000000801b9ac0a in ?? ()
(gdb) bt
#0  0x0000000801b9ac0a in ?? ()
#1  0x0000000000c75eda in mysql_init_character_set ()
#2  0x000000080191bd20 in ?? ()
#3  0x0000001f00000000 in ?? ()
#4  0x00007fffdce02120 in ?? ()
#5  0x00007fffdce02128 in ?? ()
#6  0x0000000000000012 in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x0000000000000008 in ?? ()
#9  0x00000000ffffffff in ?? ()
#10 0x00000401010ebb4c in ?? ()
#11 0x0000000000000000 in ?? ()

Best regards
Loic

Comment by Loic Quillot [ 2021-08-24 ]

Hello, here are more information, maybe it could help.

On a standalone server before upgrading to 10.5.12 :

# uname -a
FreeBSD 12.2-RELEASE-p9 FreeBSD 12.2-RELEASE-p9 r370286 amd64
 
# pkg info | grep mariadb
mariadb105-client-10.5.11      Multithreaded SQL database (client)
mariadb105-server-10.5.11      Multithreaded SQL database (server)
 
# time mysqldump --single-transaction --all-databases -p | gzip > all2.sql.gz 
Enter password: 
real    3m40.190s
user    5m9.573s 
sys     0m4.749s
 
time zcat all2.sql.gz | mysql -p
Enter password: 
real    23m43.599s
user    1m25.570s
sys     0m5.792s
 
# time zcat all2.sql.gz | mysql -p
Enter password: 
real    23m24.051s
user    1m25.286s
sys     0m6.074s
 
# time zcat all2.sql.gz | mysql -p
Enter password: 
real    23m9.138s
user    1m23.574s
sys     0m6.136s

Then after upgrading to 10.5.12 :

# service mysql-server stop 
Stopping mysql.                        
Waiting for PIDS: 35478.
 
# pkg upgrade mariadb105-server mariadb105-client
Installed packages to be UPGRADED:
        mariadb105-client: 10.5.11 -> 10.5.12
        mariadb105-server: 10.5.11 -> 10.5.12
Number of packages to be upgraded: 2
37 MiB to be downloaded.
Proceed with this action? [y/N]: y
[1/2] Fetching mariadb105-server-10.5.12.pkg: 100%   36 MiB  37.2MB/s    00:01    
[2/2] Fetching mariadb105-client-10.5.12.pkg: 100%    2 MiB   2.0MB/s    00:01    
Checking integrity... done (0 conflicting)
[1/2] Upgrading mariadb105-client from 10.5.11 to 10.5.12...
===> Creating groups.
Using existing group 'mysql'.
===> Creating users
Using existing user 'mysql'.
===> Creating homedir(s)
[1/2] Extracting mariadb105-client-10.5.12: 100%
[2/2] Upgrading mariadb105-server from 10.5.11 to 10.5.12...
===> Creating groups.
Using existing group 'mysql'.
===> Creating users
Using existing user 'mysql'.
===> Creating homedir(s)
[2/2] Extracting mariadb105-server-10.5.12: 100%
 
# service mysql-server start                                                 
Starting mysql.
 
# mysql_upgrade -p
Enter password:                                                                                
Phase 1/7: Checking and upgrading mysql database     
Processing databases                                                                           
mysql
..........
OK
 
# time zcat all2.sql.gz | mysql -p
Enter password: 
ERROR 2013 (HY000) at line 11803: Lost connection to MySQL server during query
real    17m37.162s
user    1m4.989s
sys     0m4.834s
 
#tail -70 /var/log/mysql/mysqld.err
# 2021-08-24 21:03:13 0x8e883aa00  InnoDB: Assertion failure in file /usr/ports/databases/mariadb105-server/work/mariadb-10.5.12/storage/innobase/fsp/fsp
0fsp.cc line 1535                                                                                                                                                                             
InnoDB: Failing assertion: inode                                                                                                                                                              
InnoDB: We intentionally generate a memory trap.                                                                                                                                              
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/                                                                                                                             
InnoDB: If you get repeated assertion failures or crashes, even                                                                                                                               
InnoDB: immediately after the mysqld startup, there may be                                                                                                                                    
InnoDB: corruption in the InnoDB tablespace. Please refer to                                                                                                                                  
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/                                                                                                                              
InnoDB: about forcing recovery.                                                                                                                                                               
210824 21:03:13 [ERROR] mysqld got signal 6 ;                                                                                                                                                 
This could be because you hit a bug. It is also possible that this binary                                                                                                                     
or one of the libraries it was linked against is corrupt, improperly built,                                                                                                                   
or misconfigured. This error can also be caused by malfunctioning hardware.                                                                                                                   
                                                                                                                                                                                              
To report this bug, see https://mariadb.com/kb/en/reporting-bugs                                                                                                                              
                                                                                                                                                                                              
We will try our best to scrape up some info that will hopefully help                                                                                                                          
diagnose the problem, but since we have already crashed,                                                                                                                                      
something is definitely wrong and this may fail.                                                                                                                                              
                                                                                                                                                                                              
Server version: 10.5.12-MariaDB                                                                                                                                                               
key_buffer_size=1073741824                                                                                                                                                                    
read_buffer_size=2097152                                                                                                                                                                      
max_used_connections=1                                                                                                                                                                        
max_threads=1026                                                                                                                                                                              
thread_count=1                                                                                                                                                                                
It is possible that mysqld could use up to                                                                                                                                                    
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5276648 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x8e8926fd8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fffde94bf38 thread_stack 0x49000
0x132cdfc <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
0xc75cbe <handle_fatal_signal+0x28e> at /usr/local/libexec/mariadbd
0x80191bd20 <_pthread_sigmask+0x530> at /lib/libthr.so.3
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x8e8967ff0): CREATE TABLE `sv_15` (
  `aid` int(11) NOT NULL,
  `sid` int(11) NOT NULL,
  `sdate` date NOT NULL,
  `scd` int(8) unsigned NOT NULL,
  `sct` int(8) unsigned NOT NULL, 
  `scm` int(8) NOT NULL,
  `sp` mediumint(6) NOT NULL DEFAULT 0,
  `sn` mediumint(6) NOT NULL DEFAULT 0,
  PRIMARY KEY (`aid`,`sid`,`sdate`),
  KEY `sdate` (`sdate`),
  KEY `sid` (`sid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='s/a'
 
Connection ID (thread ID): 18
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pu
shdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_ca
che=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size
=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=o
n,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: %N.core

The 10.5.11 has CVEs and we need to upgrade to 10.5.12 on tens of servers, but we don't know how to solve this issue.
Is there a way to rollback the MDEV-24393 completely to test? as setting skip_external_locking =0 in the conf.d/server.cnf didn't help.

Thank you for any solution or workaround.

Regards
Loic

Comment by IRON [ 2021-08-29 ]

It seems the problem isn't limited to 10.5 tree. 10.3 and 10.4 are also affected. See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257958

Comment by Loic Quillot [ 2021-08-30 ]

Thank you IRON, indeed.
As the MDEV-24393 also applied to these trees, do you know how to rollback this code only on the latest release, it could help me debugging.
Regards

Comment by IRON [ 2021-08-30 ]

O found only one commit regarding MDEV-24393: da094188f60bf67e3d90227304a4ea256fe2630f

The changeset is pretty simple. You can revert it by hands.

diff --git a/include/my_sys.h b/include/my_sys.h
index ac1730eeaff..1d2fff0e476 100644
--- a/include/my_sys.h
+++ b/include/my_sys.h
@@ -1,5 +1,5 @@
 /* Copyright (c) 2000, 2013, Oracle and/or its affiliates.
-   Copyright (c) 2010, 2017, MariaDB Corporation.
+   Copyright (c) 2010, 2021, MariaDB Corporation.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -279,7 +279,8 @@ extern int my_umask_dir,
 extern my_bool my_use_symdir;
 
 extern ulong   my_default_record_cache_size;
-extern my_bool  my_disable_locking, my_disable_async_io,
+extern MYSQL_PLUGIN_IMPORT my_bool my_disable_locking;
+extern my_bool  my_disable_async_io,
                 my_disable_flush_key_blocks, my_disable_symlinks;
 extern my_bool my_disable_sync, my_disable_copystat_in_redel;
 extern char    wild_many,wild_one,wild_prefix;
diff --git a/storage/innobase/os/os0file.cc b/storage/innobase/os/os0file.cc
index 884cadaf9f6..a793f9b30cb 100644
--- a/storage/innobase/os/os0file.cc
+++ b/storage/innobase/os/os0file.cc
@@ -1122,6 +1122,10 @@ os_file_lock(
        int             fd,
        const char*     name)
 {
+       if (my_disable_locking) {
+               return 0;
+       }
+
        struct flock lk;
 
        lk.l_type = F_WRLCK;

Comment by Loic Quillot [ 2021-08-30 ]

Thank you IRON, I applied the rollback on the code you posted, but I still have the same crash.
If there were no other code in the MDEV-24393 ticket, so the origin is something else.

Comment by Bernard Spil [ 2021-09-11 ]

Can you please verify if MDEV-26537 fixes this?

Currently pushing that fix for 10.5 to the FreeBSD ports tree.

Comment by Loic Quillot [ 2021-09-11 ]

Thank you Bernard,
I compiled it and installed the package on one of the server that crashed at every trial, and very good news, it didn't crash with this patch.
I will keep it running and doing the dump/inject several more times to be sure.
Thank you again
Loic

Comment by Loic Quillot [ 2021-09-11 ]

After replaying the same scenario tens of times, it didn't crash anymore.
The issue is solved here.
Thank you very much

Comment by Bernard Spil [ 2021-09-12 ]

@Loic Quillot

Note that the fix for you was 26537 and NOT this issue!

Comment by Loic Quillot [ 2021-09-12 ]

I am aware, but I don't know how to put the issue as solved...

Comment by Daniel Black [ 2021-09-20 ]

making as dup. Thanks for testing Loic

Generated at Thu Feb 08 09:44:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.