[MDEV-23115] Assertion `purge <= 1' failed in trx_rseg_mem_restore upon startup on prepared incremental backup Created: 2020-07-07  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: mariabackup, Storage Engine - InnoDB
Affects Version/s: 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 11.0
Fix Version/s: 10.5, 10.6, 10.11

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Vladislav Lesin
Resolution: Unresolved Votes: 0
Labels: None

Attachments: File mdev23115_logs.tar.gz     File mdev23115_prepared_datadir.tar.gz    

 Description   

Note: Previous versions may be affected as well, but since the datadir/backups are initially from 10.5, I cannot check earlier versions.

10.5 ee584137

mysqld: /data/src/10.5/storage/innobase/trx/trx0rseg.cc:539: void trx_rseg_mem_restore(trx_rseg_t*, trx_id_t&, mtr_t*): Assertion `purge <= 1' failed.
200707 19:53:24 [ERROR] mysqld got signal 6 ;
 
#7  0x00007f60ee164f12 in __GI___assert_fail (assertion=0x55f38a18b508 "purge <= 1", file=0x55f38a18b2d8 "/data/src/10.5/storage/innobase/trx/trx0rseg.cc", line=539, function=0x55f38a18cb60 <trx_rseg_mem_restore(trx_rseg_t*, unsigned long&, mtr_t*)::__PRETTY_FUNCTION__> "void trx_rseg_mem_restore(trx_rseg_t*, trx_id_t&, mtr_t*)") at assert.c:101
#8  0x000055f389a4cb9f in trx_rseg_mem_restore (rseg=0x55f3907ee5c8, max_trx_id=@0x7fff38bdbae0: 1735590241781178988, mtr=0x7fff38bdbbb0) at /data/src/10.5/storage/innobase/trx/trx0rseg.cc:539
#9  0x000055f389a4d080 in trx_rseg_array_init () at /data/src/10.5/storage/innobase/trx/trx0rseg.cc:617
#10 0x000055f389a5455b in trx_lists_init_at_db_start () at /data/src/10.5/storage/innobase/trx/trx0trx.cc:751
#11 0x000055f3899ffb73 in srv_start (create_new_db=false) at /data/src/10.5/storage/innobase/srv/srv0start.cc:1556
#12 0x000055f3897c9d84 in innodb_init (p=0x55f38d2e1718) at /data/src/10.5/storage/innobase/handler/ha_innodb.cc:4001
#13 0x000055f38936ef9b in ha_initialize_handlerton (plugin=0x55f38d211ff0) at /data/src/10.5/sql/handler.cc:645
#14 0x000055f389030ce3 in plugin_initialize (tmp_root=0x7fff38bdec50, plugin=0x55f38d211ff0, argc=0x55f38ab94c00 <remaining_argc>, argv=0x55f38d1d1258, options_only=false) at /data/src/10.5/sql/sql_plugin.cc:1459
#15 0x000055f389031abb in plugin_init (argc=0x55f38ab94c00 <remaining_argc>, argv=0x55f38d1d1258, flags=0) at /data/src/10.5/sql/sql_plugin.cc:1751
#16 0x000055f388ed15af in init_server_components () at /data/src/10.5/sql/mysqld.cc:4909
#17 0x000055f388ed27d1 in mysqld_main (argc=24, argv=0x55f38d1d1258) at /data/src/10.5/sql/mysqld.cc:5492
#18 0x000055f388ec7900 in main (argc=23, argv=0x7fff38be0588) at /data/src/10.5/sql/main.cc:25

To reproduce the assertion failure itself:

  • download and unpack mdev23115_prepared_datadir.tar.gz;
  • start 10.5 debug server on this datadir with

    --innodb-compression-algorithm=zlib --innodb-page-size=8K --innodb-encrypt-tables --innodb-encrypt-log --innodb-encryption-threads=4 --aria-encrypt-tables=1 --encrypt-tmp-disk-tables=1 --encrypt-binlog --file-key-management --file-key-management-filename=`pwd`/mysql-test/std_data/keys.txt --plugin-load-add=file_key_management --innodb-buffer-pool-size=512M
    

    (the path to encryption keys assumes you are starting the server from inside the basedir/sourcedir, otherwise please modify accordingly).

Non-debug build doesn't crash.


To reproduce starting from unprepared backups:

  • download and unpack ftp://ftp.askmonty.org/public/mdev23115.tar.gz. It will unpack into mdev23115 folder containing

    drwx------ 4096 backup_before_prepare_0
    drwx------ 4096 backup_before_prepare_1
    drwx------ 4096 backup_before_prepare_2
    drwx------ 4096 backup_before_prepare_3
    drwx------ 4096 backup_before_prepare_4
    drwx------ 4096 backup_before_prepare_5
    drwx------ 4096 backup_before_prepare_6
    drwx------ 4096 backup_before_prepare_7
    drwx------ 4096 backup_before_prepare_8
    drwx------ 4096 backup_before_prepare_9
    drwx------ 4096 data
    -rw-r--r-- 23036 mbackup_backup_0.log
    -rw-r--r-- 23038 mbackup_backup_1.log
    -rw-r--r-- 23088 mbackup_backup_2.log
    -rw-r--r-- 23088 mbackup_backup_3.log
    -rw-r--r-- 23137 mbackup_backup_4.log
    -rw-r--r-- 23137 mbackup_backup_5.log
    -rw-r--r-- 23145 mbackup_backup_6.log
    -rw-r--r-- 23097 mbackup_backup_7.log
    -rw-r--r-- 23097 mbackup_backup_8.log
    -rw-r--r-- 23146 mbackup_backup_9.log
    -rw-r--r-- 3205 mbackup_prepare_0.log
    -rw-r--r-- 29203 mbackup_prepare_1.log
    -rw-r--r-- 29204 mbackup_prepare_2.log
    -rw-r--r-- 29204 mbackup_prepare_3.log
    -rw-r--r-- 29204 mbackup_prepare_4.log
    -rw-r--r-- 29204 mbackup_prepare_5.log
    -rw-r--r-- 29205 mbackup_prepare_6.log
    -rw-r--r-- 29206 mbackup_prepare_7.log
    -rw-r--r-- 29206 mbackup_prepare_8.log
    -rw-r--r-- 29206 mbackup_prepare_9.log
    -rw-r--r-- 23485 mbackup_restore_.log
    -rw-r--r-- 178765 mysql.err_orig
    -rw-r----- 4792415 mysql.log
    

    the logs and the data directory are the same as in the JIRA attachments;

  • edit cnf files within mdev23115/backup_before_prepare_N folders to point at the right keys.txt file (the one in mysql-test/std_data) and at the file key management plugin;
  • run consequently

    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_1
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_2
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_3
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_4
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_5
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_6
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_7
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_8
    bin/mariabackup --prepare --use-memory=1073741824  --innodb-file-io-threads=1 --target-dir=`pwd`/mdev23115/backup_before_prepare_0 --incremental-dir=`pwd`/mdev23115/backup_before_prepare_9
    

    (modify paths to mariabackup and target/incremental dirs if needed), all prepare should go without errors;

  • try to start the server on the datadir mdev23115/backup_before_prepare_0 with

    --innodb-compression-algorithm=zlib --innodb-page-size=8K --innodb-encrypt-tables --innodb-encrypt-log --innodb-encryption-threads=4 --aria-encrypt-tables=1 --encrypt-tmp-disk-tables=1 --encrypt-binlog --file-key-management --file-key-management-filename=`pwd`/mysql-test/std_data/keys.txt --plugin-load-add=file_key_management --innodb-buffer-pool-size=512M
    



 Comments   
Comment by Alice Sherepa [ 2023-04-05 ]

also on 11.0

mysqld: /src/storage/innobase/trx/trx0rseg.cc:530: dberr_t trx_rseg_mem_restore(trx_rseg_t*, trx_id_t&, mtr_t*): Assertion `purge <= 1' failed.
230405 13:52:26 [ERROR] mysqld got signal 6 ;
 
Server version: 11.0.1-MariaDB-debug-log source revision: e599b8b45867cd24720a9ecd4ba28500f579c5a4
 
??:0(__assert_fail)[0x7f0d20638fd6]
trx/trx0rseg.cc:531(trx_rseg_mem_restore(trx_rseg_t*, unsigned long&, mtr_t*))[0x55f9e55dda2e]
trx/trx0rseg.cc:620(trx_rseg_array_init())[0x55f9e55de388]
trx/trx0trx.cc:723(trx_lists_init_at_db_start())[0x55f9e55ea88b]
srv/srv0start.cc:1542(srv_start(bool))[0x55f9e557999a]
handler/ha_innodb.cc:4176(innodb_init(void*))[0x55f9e50e750c]
sql/handler.cc:686(ha_initialize_handlerton(st_plugin_int*))[0x55f9e4720b2a]
sql/sql_plugin.cc:1465(plugin_initialize(st_mem_root*, st_plugin_int*, int*, char**, bool))[0x55f9e3e8dcff]
sql/sql_plugin.cc:1758(plugin_init(int*, char**, int))[0x55f9e3e8fc98]
sql/mysqld.cc:5235(init_server_components())[0x55f9e3a52292]
sql/mysqld.cc:5856(mysqld_main(int, char**))[0x55f9e3a54535]
sql/main.cc:34(main)[0x55f9e3a3c00d]

Generated at Thu Feb 08 09:19:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.