[MDEV-20060] Failing assertion: srv_log_file_size <= 512ULL << 30 while preparing backup Created: 2019-07-15  Updated: 2020-08-25  Resolved: 2019-08-07

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Storage Engine - InnoDB
Affects Version/s: 10.2.23
Fix Version/s: 10.2.27, 10.3.18, 10.4.8

Type: Bug Priority: Critical
Reporter: Valerii Kravchuk Assignee: Vladislav Lesin
Resolution: Fixed Votes: 1
Labels: None

Issue Links:
Problem/Incident
causes MDEV-20421 big_innodb_log reliably fails on buil... Closed

 Description   

When too much redo log is generated during backup:

...
 
[00] 2019-07-04 10:05:31 Redo log (from LSN 369453697919029 to 370244323770496) was copied.
 
[00] 2019-07-04 10:05:31 completed OK!

mariabackup --prepare stage is terminated with the assertion failure:

[00] 2019-07-05 16:28:18 mariabackup: Using 34359738368 bytes for buffer pool (set by --use-memory parameter)
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Uses event mutexes
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Using Linux native AIO
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Number of pools: 1
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Using SSE2 crc32 instructions
 
2019-07-05 16:28:18 139959018460992 [Note] InnoDB: Initializing buffer pool, total size = 32G, instances = 1, chunk size = 32G
 
2019-07-05 16:28:19 139959018460992 [Note] InnoDB: Completed initialization of buffer pool
 
2019-07-05 16:28:19 139922236876544 [Note] InnoDB: page_cleaner coordinator priority: -20
 
2019-07-05 16:28:19 0x7f4abf941740  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc line 2053
 
InnoDB: Failing assertion: srv_log_file_size <= 512ULL << 30
 
...

Note the total redo log size during backup is 736G+:

MariaDB [(none)]> select abs(369453697919029 - 370244323770496)/(1024*1024*1024) as redo_log_size;
 
+---------------+
 
| redo_log_size |
 
+---------------+
 
|      736.3277 |
 
+---------------+
 
1 row in set (0.018 sec)

It seems InnoDB code used by mariabackup is not ready to process logs larger than 512G.

 Comments   
Comment by Vladislav Lesin [ 2019-07-16 ] 

My initial intention was to create new log file during backup when the limit is reached,
 
and then just add several log files into one log group during "--prepare". But this will not work
 
as log group is considered as a tablespace with several nodes(see open_or_create_log_file())
 
and page number inside of tablespace and tablespace size are still limited with uint32_t size.
 
What I am trying to implement now is to initialize log_sys->log group with new log file when the
 
the old one is applied, and continue recovery process on "--prepare" for each log file created on
 
"--backup" step.

Comment by Vladislav Lesin [ 2019-07-31 ]

https://github.com/MariaDB/server/tree/bb-10.2-MDEV-13416-big-log
https://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.2-MDEV-13416-big-log

Comment by Vladislav Lesin [ 2019-08-01 ]

I have manually tested the above code on preparing backup with log file size greater then 512G. The following steps were used to test it:

1) Fix mariabackup to copy >512G of log file:

--- a/extra/mariabackup/xtrabackup.cc
 
+++ b/extra/mariabackup/xtrabackup.cc
 
@@ -4341,6 +4341,9 @@ static bool xtrabackup_backup_func()
 
        datafiles_iter_free(it);
 
        }
 
 
+       lsn_t to_copy = checkpoint_lsn_start + 800000000000;
 
+       backup_wait_for_lsn(to_copy);
 
+
 
        bool ok = backup_start();
 
 
        if (ok) {

2) Build the server with default options:

cmake ../mariadb-server/ -DBUILD_CONFIG=mysql_release

3) Init data directory and start the server:

./scripts/mysql_install_db --no-defaults --srcdir=/work/mariadb-server --datadir=/work/data --plugin-dir=/work/build/plugin --lc-messages-dir=/work/build/sql/share/ --character-sets-dir=/work/mariadb-server/sql/share/charsets --innodb_log_file_size=1000000000 --innodb_file_per_table=1
 
sql/mysqld --no-defaults --basedir=/work/mariadb-server --datadir=/work/data --plugin-dir=/work/build/plugin --lc-messages-dir=/work/build/sql/share/ --character-sets-dir=/work/mariadb-server/sql/share/charsets --innodb_log_file_size=1000000000 --innodb-flush-log-at-trx-commit=0 --innodb-buffer-pool-size=10G --innodb-file-per-table --log-bin=false --max-prepared-stmt-count=1048576 --gdb

4) Load the server with sysbench oltp_write_only test:

./src/sysbench --threads=56 --time=0 --report-interval=5 --mysql-user=root --mysql-db=test --tables=100 --table_size=100000 oltp_write_only prepare
 
./src/sysbench --threads=56 --time=0 --report-interval=5 --mysql-user=root --mysql-db=test --tables=100 --table_size=100000 oltp_write_only run

5) Backup, prepare

./extra/mariabackup/mariabackup --backup --user=root --target-dir=/work/backup
 
./extra/mariabackup/mariabackup --prepare --target-dir=/work/backup

6) Restart the server on backed up data directory:

sql/mysqld --no-defaults --basedir=/work/mariadb-server --datadir=/work/backup --plugin-dir=/work/build/plugin --lc-messages-dir=/work/build/sql/share/ --character-sets-dir=/work/mariadb-server/sql/share/charsets --innodb_log_file_size=1000000000 --innodb-flush-log-at-trx-commit=0 --innodb-buffer-pool-size=10G --innodb-file-per-table --log-bin=false --max-prepared-stmt-count=1048576 --gdb

There were not errors during "backup --prepare" and server restart.

Generated at Thu Feb 08 08:56:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.