[#MDEV-29789] InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page

[MDEV-29789] InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page) Created: 2022-10-14 Updated: 2022-10-14 Resolved: 2022-10-14
Status:	Closed
Project:	MariaDB Server
Component/s:	Storage Engine - InnoDB
Affects Version/s:	10.1.14
Fix Version/s:	N/A

Type:

Bug

Priority:

Major

Reporter:

thejaraj

Assignee:

Daniel Black

Resolution:

Won't Fix

Votes:

Labels:

None

Environment:

Server version: 10.1.14-MariaDB

Description

Team, MySQL is getting crashing again and again due to below error log, kindly let me know how to fix teh issues
i have already changed innodb_force_recovery = 1, but still facing issues

InnoDB: for more information.
2022-10-13 14:29:27 7f212effc700 InnoDB: Assertion failure in thread 139780499162880 in file btr0cur.cc line 332
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
221013 14:29:27 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.14-MariaDB
key_buffer_size=268435456
read_buffer_size=2097152
max_used_connections=2
max_threads=302
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1505301 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55816a85a1be]
/usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x55816a385f0d]
/lib64/libpthread.so.0(+0xf100)[0x7f3b4b547100]
/lib64/libc.so.6(gsignal+0x37)[0x7f3b4961b5f7]
/lib64/libc.so.6(abort+0x148)[0x7f3b4961cce8]
/usr/sbin/mysqld(+0x857953)[0x55816a62c953]
/usr/sbin/mysqld(+0x859a5b)[0x55816a62ea5b]
/usr/sbin/mysqld(+0x867de6)[0x55816a63cde6]
/usr/sbin/mysqld(+0x7ee4e4)[0x55816a5c34e4]
/usr/sbin/mysqld(+0x7f0511)[0x55816a5c5511]
/usr/sbin/mysqld(+0x7b7167)[0x55816a58c167]
/usr/sbin/mysqld(+0x81dcbe)[0x55816a5f2cbe]
/usr/sbin/mysqld(+0x80bcc6)[0x55816a5e0cc6]
/lib64/libpthread.so.0(+0x7dc5)[0x7f3b4b53fdc5]
/lib64/libc.so.6(clone+0x6d)[0x7f3b496dc28d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
^C
[root@ins0036 mysql01]$ cat /etc/my.cnf
[mysqld_safe]
pid-file=/var/run/mariadb/mariadb.pid

include all files from the config directory
#
#!includedir /etc/my.cnf.d
[mysqld]
datadir = /mnt/spool/mysql01
tmpdir = /mnt/spool/mysql-tmp
#tmpdir = /dev/shm/mysql-tmp
socket = /var/lib/mysql/mysql.sock
log_error = /var/log/mariadb/mariadb.log
user = mysql
skip-external-locking
port = 3306
max_allowed_packet = 64M
max_connections = 300

buffers and caches
key_buffer_size = 256M
table_cache = 4096
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
join_buffer_size = 2M
thread_cache_size = 64
query_cache_size = 16G
query_cache_type = 1
query_cache_limit = 16M
myisam_sort_buffer_size = 1M
#optimizer_switch = 'index_condition_pushdown=off'

#innodb
innodb_file_per_table=1
innodb_buffer_pool_size = 80G
innodb_buffer_pool_instances = 64
innodb_log_buffer_size = 8M
innodb_log_file_size = 1G
innodb_log_files_in_group = 2
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 1
innodb_force_recovery = 0
innodb_fast_shutdown=0
#innodb_purge_threads=0
#innodb_use_native_aio = 0
innodb-write-io-threads=16

#innodb_lock_wait_timeout = 50
server-id = 1

MASTER
log-bin = master-bin
relay_log = relay-bin
binlog-format = mixed
sync-binlog = 1
expire_logs_days = 3

SLAVE
#read-only = 1
#log_bin = slave-bin
#relay_log = relay-bin
#binlog-format = mixed
#sync-binlog = 1
#expire_logs_days = 5
#log-slave-updates = 1
#replicate_ignore_db = cacti

#slow_query_log
#slow_query_log_file = /mnt/spool/mariadb/mariadb-slow.log
#long_query_time = 2

free -g
total used free shared buff/cache available
Mem: 125 2 0 2 123 120
Swap: 1 0 1

Comments

Comment by Daniel Black [ 2022-10-14 ]

10.1 and 10.2 release series are currently out of maintenance - https://mariadb.org/about/#maintenance-policy.

The assertion was replaced with error handling in ~~MDEV-13542~~. That is, the server should no longer crash when it encounters this type of corruption.

While a 10.1/10.2 fix isn't going to happen, details may help indicate if this is potentially still a bug in later versions, do you know anything that change or occurred recently potentially related to the crashed table? What is the oldest version of MySQL or MariaDB with which the database was created? What methods of backup and restore (or copying of database files in binary form) are being used? Has the configuration parameter innodb_force_recovery ever been used beyond the =1 described in this bug report?

Comment by Marko Mäkelä [ 2022-10-14 ]

Setting innodb_force_recovery=1 (which is mentioned in the Description) could actually cause this problem. If the redo log would have covered splitting or merging two index B-tree pages and the changes to one of the pages were discarded due to innodb_force_recovery, the "next" and "previous" sibling links between the pages would trivially become out of sync.

In 10.1, recovery was much more prone to errors. The situation was improved already in 10.2 by ~~MDEV-12699~~ and some follow-up fixes.

Comment by thejaraj [ 2022-10-14 ]

HI Daniel,

we could see bulk insert operation in recent logs, and we have update the parameter innodb_force_recovery=1, but still maradb is crashing contunusoly,

we have slave node we are taking the file level backup from networker and restore on master

Comment by thejaraj [ 2022-10-14 ]

could you please help me with the steps to restore form slave server to master server

Comment by Daniel Black [ 2022-10-14 ]

An offline copy of the datadir from slave to master should be sufficient. This kind of assistance is beyond what can be offered reliably in a bug report ticket.

Please look at doing an upgrade soon. Keen in mind slave servers need to be the same or greater version than the master server. Some variables are now deprecated with no replacement (innodb_buffer_pool_instances at least).

Comment by thejaraj [ 2022-10-14 ]

when i'm trying to restore from slave server, i'm getting below error, kindly check and update on below issues

2022-10-14 08:17:57 7ef62fffc700 InnoDB: Error: page 3646512 log sequence number 11882443412821
InnoDB: is in the future! Current system log sequence number 11882423160976.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
/lib64/libc.so.6(abort+0x148)[0x7f0fd6ec1ce8]
/usr/sbin/mysqld(+0x7a0c20)[0x559b39b67c20]
2022-10-14 08:17:57 7ef62fffc700 InnoDB: Error: page 2981888 log sequence number 11882437570369
InnoDB: is in the future! Current system log sequence number 11882423160976.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
/usr/sbin/mysqld(+0x79fab5)[0x559b39b66ab5]
/usr/sbin/mysqld(+0x7a032b)[0x559b39b6732b]
/usr/sbin/mysqld(+0x77b84f)[0x559b39b4284f]
/usr/sbin/mysqld(+0x77dd1f)[0x559b39b44d1f]
/usr/sbin/mysqld(+0x885c85)[0x559b39c4cc85]
/usr/sbin/mysqld(+0x8db3f8)[0x559b39ca23f8]
2022-10-14 08:17:57 7ef62fffc700 InnoDB: Error: page 908646 log sequence number 11882488872927
InnoDB: is in the future! Current system log sequence number 11882423160976.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
/usr/sbin/mysqld(+0x80e750)[0x559b39bd5750]
/lib64/libpthread.so.0(+0x7dc5)[0x7f0fd8de4dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f0fd6f8128d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by Daniel Black [ 2022-10-14 ]

was the master server entirely stopped (no mariadb running) and the datadir empty before you copied from an offline (no mariadb running) slave's datadir to this location?

Comment by thejaraj [ 2022-10-14 ]

master server is stopped, and keep crashing with below error, i have changed the new path location of data dir files copied form slave server, on master node

let me know how to fix this below issues.

InnoDB: for more information.
2022-10-14 08:28:38 7ef496ffc700 InnoDB: Error: page 4172 log sequence number 11882424683467
InnoDB: is in the future! Current system log sequence number 11882423160976.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-14 08:28:38 7ef496ffc700 InnoDB: Error: page 2762306 log sequence number 11882437664701
InnoDB: is in the future! Current system log sequence number 11882423160976.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
/lib64/libpthread.so.0(+0x7dc5)[0x7f10f5481dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f10f361e28d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by thejaraj [ 2022-10-14 ]

Team, you help is needed to fix the issues, after updating the below parameter i'm facing issues, kindly help
default-storage-engine=MyISAM

MariaDB [zabbix]> select count(1) from triggers;
ERROR 1286 (42000): Unknown storage engine 'InnoDB'
MariaDB [zabbix]>
login as: nxf85371

Comment by Daniel Black [ 2022-10-14 ]

This kind of assistance is beyond what can be offered in a bug report ticket.

This isn't a help forum - see https://mariadb.com/kb/en/getting-help-with-mariadb/ for options.

Generated at Thu Feb 08 10:11:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MDEV-29789] InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page) Created: 2022-10-14 Updated: 2022-10-14 Resolved: 2022-10-14