[MDEV-12988] backup fails if innodb_undo_tablespaces>0 Created: 2017-06-03  Updated: 2021-11-24  Resolved: 2017-08-17

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1.24
Fix Version/s: 10.1.27, 10.2.9, 10.3.2

Type: Bug Priority: Major
Reporter: Michael Xu Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7.x x86_64


Issue Links:
Duplicate
is duplicated by MDEV-13243 MariaBackup failed, while percona-xtr... Closed
is duplicated by MDEV-13305 InnoDB: Assertion failure in file /ho... Closed
Relates
relates to MDEV-13560 restart_and_restore.inc does not pass... Closed
relates to MDEV-13561 Mariabackup is incompatible with retr... Closed
relates to MDEV-27121 mariabackup incompatible with disable... Closed

 Description   

# mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
Using server version 10.1.24-MariaDB
mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64) 
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /opt/mysql
xtrabackup: open files limit requested 65535, set to 100001
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = /opt/mysql
xtrabackup:   innodb_data_file_path = ibdata1:128M:autoextend
xtrabackup:   innodb_log_group_home_dir = /opt/mysql
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 268435456
2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
xtrabackup: using O_DIRECT
xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
170603 08:21:55 >> log scanned up to (376182854)

2017-06-03 08:21:55 7f1f34a25900  InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
170603  8:21:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.1.24-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
addr2line: 'mariabackup': No such file
mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
/lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
/lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
/lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
mariabackup(+0x91862b)[0x7f1f3538362b]
mariabackup(+0x3a7de4)[0x7f1f34e12de4]
mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
mariabackup(main+0xb4d)[0x7f1f34df899d]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
mariabackup(+0x3a624d)[0x7f1f34e1124d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Aborted

Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.



 Comments   
Comment by Vladislav Vaintroub [ 2017-06-13 ]

Please share your my.cnf file please.

Comment by Andrii Nikitin (Inactive) [ 2017-06-14 ]

I can repeat the problem with 10.1.24 and latest 10.1 build by calling mysql_install_db with innodb_undo_tablespaces=4 .

10.1.24/bin/mariabackup --defaults-file=my.cnf --target-dir=bkup --backup
170614 17:44:06 Connecting to MySQL server host: localhost, user: root, password: not set, port: 3313, socket: /home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
Using server version 10.1.24-MariaDB
../_depot/m-tar/10.1.24/bin/mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64) 
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /home/a/test1/mariadb-environs/m7-10.1.24/dt
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 50331648
170614 17:44:06 >> log scanned up to (1501756)
2017-06-14 17:44:06 7f622943b640  InnoDB: Assertion failure in thread 140059575825984 in file srv0start.cc line 1475
InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]

my.cnf

[xtrabackup]
user=root
port=3313
 
[client]
user=root
port=3313
socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
 
[mysqld]
server_id=7
port=3313
socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
datadir=/home/a/test1/mariadb-environs/m7-10.1.24/dt
log-error=/home/a/test1/mariadb-environs/m7-10.1.24/dt/error.log
 
pid_file=/home/a/test1/mariadb-environs/m7-10.1.24/dt/p.id
 
lc_messages_dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/share
plugin-dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/lib/plugin
innodb_undo_tablespaces=4

10.1.23 doesn't seem to be affected.

Comment by Andrii Nikitin (Inactive) [ 2017-07-13 ]

10.2.7 shows little different assert

2017-07-13 15:10:40 0x7f3d70464740  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc line 960
InnoDB: Failing assertion: undo_tablespace_ids[i] != 0
InnoDB: We intentionally generate a memory trap.

Comment by Michael Xu [ 2017-07-14 ]

my.cnf attached, and I think the error may caused by undo_tablespaces setting

# this is only for the mysqld standalone daemon
[mysqld]
innodb_undo_directory                 = /var/lib/mysql                 # 
innodb_undo_tablespaces               = 12                             # 
innodb_undo_logs                      = 128                            # 
innodb_max_undo_log_size              = 1G                             # 
innodb_undo_log_truncate              = 1                              #
[mariadb]
[mariadb-10.1]

Comment by Marko Mäkelä [ 2017-08-10 ]

MariaDB 10.1.24 was the first one to contain the merge from MySQL 5.6.36, which introduced
Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES
which is likely causing this issue. We must adapt Mariabackup to this change.

Comment by Marko Mäkelä [ 2017-08-17 ]

I can trivially repeat with

./mtr --mysqld=--innodb-undo-tablespaces=4 --suite=mariabackup

on both 10.1 and 10.2.

Comment by Marko Mäkelä [ 2017-08-17 ]

The problem is that in backup mode, srv_undo_tablespaces_init() will not initialize the undo_tablespace_ids[]. In normal startup, trx_rseg_get_n_undo_tablespaces() would take care of it. During backup, we do not have a buffer pool, and we cannot access the TRX_SYS page directly in the system tablespace, because the file page could be stale (some changes for it could be in the redo log only). Because backup does not really parse individual redo log records, and because we need to discover all tablespaces (including the undo tablespaces) at the start of the backup, some other mechanism is needed.

As of today, Percona Xtrabackup 2.3 is still based on MySQL 5.6.14, so it is missing any adjustment for the MySQL 5.6.36 change
Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES
But, it does initialize the undo_tablespace_ids[] to a sequence 1,2,3… (should be srv_undo_space_id_start,srv_undo_space_id_start+1,… with the above change).

Mariabackup should read the first page of the undo001 file and determine the tablespace ID from there. Perhaps it could simply treat all the undo* files in the same way as *.ibd files.

Comment by Marko Mäkelä [ 2017-08-17 ]

To reduce the scope of this fix, I filed a separate bug for the 5.6.36 incompatibility:
MDEV-13561 Mariabackup is incompatible with retroactively created innodb_undo_tablespaces

Generated at Thu Feb 08 08:02:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.