[MDEV-29780] Maria DB goes down with error InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.5/storage/innobase/btr/btr0pcur.cc line 512 Created: 2022-10-12  Updated: 2022-10-19  Resolved: 2022-10-19

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.6.5
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Francisco Ortiz B. Assignee: Daniel Black
Resolution: Incomplete Votes: 0
Labels: crash
Environment:

Linux DBServer0001 3.10.0-1062.1.2.el7.x86_64 #1 SMP Fri Sep 20 10:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux



 Description   

Maria DB instance goes down throwing the following:

2022-10-12 14:13:27 0x7f5a780b0700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.5/storage/innobase/btr/btr0pcur.cc line 512
InnoDB: Failing assertion: btr_page_get_prev(next_page) == btr_pcur_get_block(cursor)->page.id().page_no()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
221012 14:13:27 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.6.5-MariaDB-log
key_buffer_size=7516192768
read_buffer_size=67108864
max_used_connections=22
max_threads=1002
thread_count=22
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 138700241 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
 
Thread pointer: 0x7f4db0000b88
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f5a780afcc0 thread_stack 0x49000
Can't start addr2line
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55a33a13aa4e]
/usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x55a339b8ae27]
/lib64/libpthread.so.0(+0xf630)[0x7f5d50f10630]
/lib64/libc.so.6(gsignal+0x37)[0x7f5d5035c377]
/lib64/libc.so.6(abort+0x148)[0x7f5d5035da68]
/usr/sbin/mariadbd(+0x668f13)[0x55a339845f13]
/usr/sbin/mariadbd(+0xe0499c)[0x55a339fe199c]
/usr/sbin/mariadbd(+0xd7d0c0)[0x55a339f5a0c0]
/usr/sbin/mariadbd(+0xc94655)[0x55a339e71655]
/usr/sbin/mariadbd(_ZN7handler13ha_index_nextEPh+0x16f)[0x55a339b9143f]
/usr/sbin/mariadbd(+0x7c9c4c)[0x55a3399a6c4c]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x295)[0x55a33999cd65]
/usr/sbin/mariadbd(_ZN4JOIN10exec_innerEv+0xf4d)[0x55a3399cc12d]
/usr/sbin/mariadbd(_ZN4JOIN4execEv+0x33)[0x55a3399cc573]
/usr/sbin/mariadbd(_ZN18st_select_lex_unit4execEv+0x73c)[0x55a339a15c4c]
/usr/sbin/mariadbd(+0x74b732)[0x55a339928732]
/usr/sbin/mariadbd(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0xbd)[0x55a33992819d]
/usr/sbin/mariadbd(_ZN13st_join_table12preread_initEv+0x79)[0x55a33999c9a9]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x1b8)[0x55a33999cc88]
/usr/sbin/mariadbd(+0x7ba0cc)[0x55a3399970cc]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x1ff)[0x55a33999cccf]
/usr/sbin/mariadbd(_ZN4JOIN10exec_innerEv+0xf4d)[0x55a3399cc12d]
/usr/sbin/mariadbd(_ZN4JOIN4execEv+0x33)[0x55a3399cc573]
/usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x15e)[0x55a3399ca60e]
 
 
/usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1e4)[0x55a3399cae74]
/usr/sbin/mariadbd(_ZN25Sql_cmd_create_table_like7executeEP3THD+0x586)[0x55a339a04d76]
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x1f9a)[0x55a33996cfba]
/usr/sbin/mariadbd(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x38)[0x55a3398c3d38]
/usr/sbin/mariadbd(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x1be)[0x55a3398ccc5e]
/usr/sbin/mariadbd(_ZN13sp_instr_stmt7executeEP3THDPj+0x5de)[0x55a3398cd71e]
/usr/sbin/mariadbd(_ZN7sp_head7executeEP3THDb+0xb04)[0x55a3398c7cb4]
/usr/sbin/mariadbd(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x7af)[0x55a3398c95ef]
/usr/sbin/mariadbd(+0x7837ca)[0x55a3399607ca]
/usr/sbin/mariadbd(_ZN12Sql_cmd_call7executeEP3THD+0x17a)[0x55a339964faa]
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x1f9a)[0x55a33996cfba]
/usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x20b)[0x55a3399709db]
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1217)[0x55a339972c27]
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x123)[0x55a3399742c3]
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x187)[0x55a339a68377]
/usr/sbin/mariadbd(handle_one_connection+0x34)[0x55a339a68614]
/usr/sbin/mariadbd(+0xbedbb2)[0x55a339dcabb2]
/lib64/libpthread.so.0(+0x7ea5)[0x7f5d50f08ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f5d504248cd]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.



 Comments   
Comment by Daniel Black [ 2022-10-12 ]

Do you have the CREATE TABLE LIKE sql that triggered this? And the SHOW CREATE TABLE tablename for the table you are copying off?

Comment by Daniel Black [ 2022-10-12 ]

This also looks like MDEV-28002 and is probably a duplicate of MDEV-13542.

Comment by Marko Mäkelä [ 2022-10-13 ]

The assertion was replaced with error handling in MDEV-13542. That is, the server should no longer crash when it encounters this type of corruption.

How could the corruption have happened? What is the oldest version of MySQL or MariaDB with which the database was created? What methods of backup and restore (or copying of database files in binary form) are being used? Has the configuration parameter innodb_force_recovery ever been used?

Comment by Francisco Ortiz B. [ 2022-10-13 ]

Hello dears,

Thank you for your answers. For security reasons I can not add the code of the instruction, but if
necessary I can send it if they tell me some internal mail.
The database was migrated from version 10.4.16 to version 10.6.5 of Maria DB.
The backups are made via mariabackup, but only on Sundays, the fall was suffered on Wednesday.
The innodb_force_recover parameter has not been used.
Reviewing the indicated notes, I will make an OPTIMEZE TABLE of the table involved. And it will also be disabled
the innodb_change_buffering parameter.
If the problem persist, we will analyze migrate to MariaDB version at least 10.6.9.

Regards,
Francisco.

Comment by Daniel Black [ 2022-10-19 ]

Thanks for the offer of sending information. Without the steps that got to this state we have doubt we can make use of it.

Your upgrade plan sounds good but do keep in mind that 10.6.10 has some important fixes and a 10.6.11 release is currently being prepared.

If you encounter this or any other issue after your upgrade and have good details as to how this occurred we'll gladly take a deeper look.

Thanks for using MariaDB and reporting bugs.

Generated at Thu Feb 08 10:11:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.