[MDEV-31321] regular crashes after upgrade from 10.6.12-1 to 10.6.13-1 Created: 2023-05-21  Updated: 2023-07-23  Resolved: 2023-07-23

Status: Closed
Project: MariaDB Server
Component/s: OTHER
Affects Version/s: 10.6.13
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Zep Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

centos7 x64


 Description   

Hello

After upgrading from 10.6.12-1 to 10.6.13-1 we are seeing regular crashes on a specific server.

Thread pointer: 0x7fd31c0009f8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7fd374163c90 thread_stack 0x20000
 
??:0(my_print_stacktrace)[0x55fbbfbe7e3e]
 
??:0(handle_fatal_signal)[0x55fbbf5d99c7]
 
sigaction.c:0(__restore_rt)[0x7fd37f404630]
 
:0(__GI_raise)[0x7fd37e84f387]
 
:0(__GI_abort)[0x7fd37e850a78]
 
/usr/sbin/mariadbd(+0x66b98d)[0x55fbbf2a998d]
 
??:0(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55fbbf9a60bf]
 
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x55fbbf8f58a7]
 
/usr/sbin/mariadbd(+0x655889)[0x55fbbf293889]
 
??:0(mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool))[0x55fbbf46ecb3]
 
??:0(mysql_recreate_table(THD*, TABLE_LIST*, bool))[0x55fbbf46fd6e]
 
??:0(MDL_lock::lf_alloc_constructor(unsigned char*))[0x55fbbf4d1e75]
 
??:0(MDL_lock::lf_alloc_constructor(unsigned char*))[0x55fbbf4d3ae1]
 
??:0(Sql_cmd_optimize_table::execute(THD*))[0x55fbbf4d4dc3]
 
??:0(mysql_execute_command(THD*))[0x55fbbf3d09c7]
 
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55fbbf3d52e5]
 
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55fbbf3d7bb5]
 
??:0(do_command(THD*))[0x55fbbf3d934f]
 
??:0(do_handle_one_connection(CONNECT*, bool))[0x55fbbf4c6bc2]
 
??:0(handle_one_connection)[0x55fbbf4c6e94]
 
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x55fbbf82f1b7]
 
pthread_create.c:0(start_thread)[0x7fd37f3fcea5]
 
??:0(__clone)[0x7fd37e917b0d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7fd31c090f40): OPTIMIZE TABLE `innodb_index_stats`
 
tack_bottom
 
Connection ID (thread ID): 159339
 
Status: NOT_KILLED
 
 
 
 
 
Thread pointer: 0x7f242c0008d8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f24900a2c90 thread_stack 0x20000
 
??:0(my_print_stacktrace)[0x5566412bba6e]
 
??:0(handle_fatal_signal)[0x556640cb01a7]
 
sigaction.c:0(__restore_rt)[0x7f2498b01630]
 
:0(__GI_raise)[0x7f2497f4c387]
 
:0(__GI_abort)[0x7f2497f4da78]
 
/usr/sbin/mariadbd(+0x66ece5)[0x55664097dce5]
 
??:0(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x55664107958f]
 
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x556640fcaaef]
 
??:0(mysql_alter_table(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool, bool))[0x556640b432f3]
 
??:0(mysql_recreate_table(THD*, TABLE_LIST*, bool))[0x556640b443ae]
 
??:0(MDL_lock::lf_alloc_constructor(unsigned char*))[0x556640ba6cc5]
 
??:0(MDL_lock::lf_alloc_constructor(unsigned char*))[0x556640ba89a1]
 
??:0(Sql_cmd_optimize_table::execute(THD*))[0x556640ba9cd3]
 
??:0(mysql_execute_command(THD*))[0x556640aa6326]
 
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x556640aaa9f5]
 
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x556640aad255]
 
??:0(do_command(THD*))[0x556640aae9f7]
 
??:0(do_handle_one_connection(CONNECT*, bool))[0x556640b9ba12]
 
??:0(handle_one_connection)[0x556640b9bce4]
 
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x556640f076d7]
 
pthread_create.c:0(start_thread)[0x7f2498af9ea5]
 
??:0(__clone)[0x7f2498014b0d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f242c022350): OPTIMIZE TABLE `innodb_index_stats`
 
 
 
Connection ID (thread ID): 264257
 
Status: NOT_KILLED

Trying to repair/optimize tables doesn't work. We go back to 10.6.12 and no issues.

We try the upgrade 3 times without and after a few hours we see the regular crashes.

Thank you

 Comments   
Comment by Daniel Black [ 2023-05-21 ]

Apparently not fixed in MDEV-25919 per MDEV-23670.

Was a core dump created? If so can you extract a backtrace?

Can you, even on the 10.6.12 version, extract the contents of innodb_index_stats? mariadb-dump mysql innodb_index_stats > /tmp/index.sql. If you don't want to show your table names, can you ftp upload privately or change the database/table names in the SQL file

Comment by Zep [ 2023-05-22 ]

Hello

This is a production server and also core dumps are off. I can clone the server, upgrade mariadb and enable core dumps if need but it will take me some time.

I have upload index stats from 10.6.12.

Thank you

Comment by Daniel Black [ 2023-05-22 ]

Thanks for MDEV-31321.tar and the offer. I'll see if I can reproduce the bug with this before your attempt at cloning/reproduction.

Comment by Zep [ 2023-06-16 ]

Hello,
10.6.14-1 as been released and we try the upgrade from 10.6.12-1 >> 10.6.14-1.
After a few days everything seems to be working just fine.
Thank you!

Comment by Elena Stepanova [ 2023-07-23 ]

Thanks for the update.

I'll close it as "Cannot reproduce" for the lack of a suitable resolution – since we don't know what exactly fixed it, we can't claim it was fixed.

Generated at Thu Feb 08 10:22:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.