[#MDEV-18325] "InnoDB: Error: page ... log sequence number ... is in the future!" upon startup after restoring from incremental backup

Note: The test flow here is supposed to be DML-only (after the initial table creation). In order to verify it, the general log is in the archive along with backup logs and datadirs.

10.1 1d72db45a8
recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 89% 99% 100% (19.4 seconds); tables to flush: 9 8 7 6 5 4 3 2 1 0 (0.0 seconds);
2019-01-20 19:41:40 140187119347520 [Warning] InnoDB: New log files created, LSN=1618875
2019-01-20 19:41:40 7f7fdb736740 InnoDB: Error: page 5 log sequence number 1619059
InnoDB: is in the future! Current system log sequence number 1618956.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2019-01-20 19:41:41 140187119347520 [Warning] /home/travis/server/bin/mysqld: unknown variable 'loose-debug_assert_on_not_freed_memory=0'
Version: '10.1.38-MariaDB-debug' socket: '/home/travis/logs/vardir/mysql.sock' port: 19300 Source distribution

The backup datadirs before prepare, mariabackup logs and server logs can be found here:
ftp://ftp.askmonty.org/public/mdev18325.tar.gz

To reproduce, unpack backup_before_prepare_* folders, adjust the path to the plugin dir and the encryption key file and run incremental prepare (backup_before_prepare_0 is the initial full backup, backup_before_prepare_1 and backup_before_prepare_2 are incremental backups):

bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=backup_before_prepare_0

bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=backup_before_prepare_0 --incremental-dir=backup_before_prepare_1

bin/mariabackup --prepare --apply-log-only --innodb-file-io-threads=1 --target-dir=backup_before_prepare_0 --incremental-dir=backup_before_prepare_2

and then start the server with

--innodb-encrypt-tables --innodb-encrypt-log --innodb-encryption-threads=4 --aria-encrypt-tables=1 --encrypt-tmp-disk-tables=1 --file-key-management --file-key-management-filename=/data/bld/keys.txt  --plugin-load-add=file_key_management

(adjust the path to the key file as needed).

I don't know whether there would be any further effect on InnoDB side, because all test tables are actually Aria (and CHECK tables on them returns errors, but it's probably unrelated to this InnoDB issue.)

I have no information yet whether it affects higher versions.

Notes on the origin of the report

Please note it's not necessarily reproducible this way.

https://travis-ci.org/elenst/travis-tests/jobs/481813571 (Trial 2)
Server: 10.1 1d72db45a880d07fec5eda648601235f96a693c0
Tests: elenst-jira-refs 19e81706fbfffc28d2f71d2c22b361ecd4593d9a
Toolbox: dbef832b2c61379aa92cad46bd53159b766d3b3e

perl ./runall-new.pl --basedir=/home/travis/server --vardir=/home/travis/logs/vardir --duration=350 --threads=4 --seed=1548013050 --reporters=Backtrace,ErrorLog,Deadlock --skip-gendata --gendata-advanced --views --grammar=conf/mariadb/generic-dml.yy --redefine=conf/mariadb/bulk_insert.yy --mysqld=--log_output=FILE --mysqld=--max-statement-time=20 --mysqld=--lock-wait-timeout=10 --mysqld=--loose-innodb-lock-wait-timeout=5 --mysqld=--loose-debug_assert_on_not_freed_memory=0 --filter=/home/travis/mariadb-toolbox/travis/10.4-combo-filter.ff --mysqld=--innodb-encrypt-tables --mysqld=--innodb-encrypt-log --mysqld=--innodb-encryption-threads=4 --mysqld=--aria-encrypt-tables=1 --mysqld=--encrypt-tmp-disk-tables=1 --mysqld=--file-key-management --mysqld=--file-key-management-filename=/home/travis/mariadb-toolbox/data/keys.txt --mysqld=--plugin-load-add=file_key_management --engine=Aria --scenario=MariaBackupIncremental

[MDEV-18325] "InnoDB: Error: page ... log sequence number ... is in the future!" upon startup after restoring from incremental backup Created: 2019-01-21 Updated: 2021-05-11
Status:	Open
Project:	MariaDB Server
Component/s:	mariabackup, Storage Engine - InnoDB
Affects Version/s:	10.1
Fix Version/s:	10.1

[MDEV-18325] "InnoDB: Error: page ... log sequence number ... is in the future!" upon startup after restoring from incremental backup Created: 2019-01-21 Updated: 2021-05-11