[MDEV-26794] MariaBackup does not recognize added providers upon prepare of incremental backup Created: 2021-10-09  Updated: 2021-10-27  Resolved: 2021-10-22

Status: Closed
Project: MariaDB Server
Component/s: mariabackup, Plugins
Affects Version/s: N/A
Fix Version/s: 10.7.1

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Problem/Incident
is caused by MDEV-12933 sort out the compression library chaos Closed

 Description   

It looks like upon prepare of an incremental backup MariaBackup reads options (at least plugins) from the config file stored in the original full backup directory. So, if a provider was added since the original backup, MariaBackup cannot prepare tables compressed with the added algorithms.

The MTR test case below simply hangs upon the 2nd prepare. If the output is redirected to a log, the last lines after which mariabackup hangs are

preview-10.7-MDEV-12933-provider-plugins e0fee003

 
2021-10-10  0:15:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=49872,49872
 
2021-10-10  0:15:44 0 [Note] InnoDB: Starting final batch to recover 6 pages from redo log.
 
2021-10-10  0:15:44 0 [Warning] InnoDB: 'test/t_snappy.ibd' is compressed with snappy, which is not currently loaded

and the stack trace is 

Thread 1 (Thread 0x7ff4e0610800 (LWP 3598619)):
 
#0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5605dab74cf8 <recv_sys+216>) at ../sysdeps/nptl/futex-internal.h:183
 
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5605dab74c48 <recv_sys+40>, cond=0x5605dab74cd0 <recv_sys+176>) at pthread_cond_wait.c:508
 
#2  __pthread_cond_wait (cond=0x5605dab74cd0 <recv_sys+176>, mutex=0x5605dab74c48 <recv_sys+40>) at pthread_cond_wait.c:638
 
#3  0x00005605da03def2 in safe_cond_wait (cond=0x5605dab74cd0 <recv_sys+176>, mp=0x5605dab74c20 <recv_sys>, file=0x5605da4f1ba0 "/data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/log/log0recv.cc", line=3143) at /data/src/preview-10.7-MDEV-12933-provider-plugins/mysys/thr_mutex.c:492
 
#4  0x00005605d9d4ce7d in recv_sys_t::apply (this=0x5605dab74c20 <recv_sys>, last_batch=true) at /data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/log/log0recv.cc:3143
 
#5  0x00005605d9e8e001 in srv_start (create_new_db=false) at /data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/srv/srv0start.cc:1488
 
#6  0x00005605d9216e06 in innodb_init () at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:2144
 
#7  0x00005605d922290c in xtrabackup_prepare_func (argv=0x5605dba614c8) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:5891
 
#8  0x00005605d92252f8 in main_low (argv=0x5605dba614c8) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:6793
 
#9  0x00005605d9224b29 in main (argc=4, argv=0x7ffd3c3ff558) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:6591


--source include/have_innodb.inc
 
 
 
--let $targetdir=$MYSQLTEST_VARDIR/tmp/backup
 
--let $incdir=$MYSQLTEST_VARDIR/tmp/inc
 
 
 
CREATE TABLE t (a INT) ENGINE=InnoDB;
 
INSERT INTO t VALUES (1),(2);
 
 
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --target-dir=$targetdir
 
 
 
INSTALL SONAME 'provider_snappy';
 
 
 
SET GLOBAL innodb_compression_algorithm= snappy;
 
 
 
CREATE TABLE t_snappy (a INT) ENGINE=InnoDB page_compressed=1;
 
INSERT INTO t_snappy VALUES (3),(4);
 
 
 
--source include/restart_mysqld.inc
 
 
 
--exec $XTRABACKUP --defaults-file=$MYSQLTEST_VARDIR/my.cnf --backup --incremental-basedir=$targetdir --target-dir=$incdir
 
 
 
--echo # Prepare initial backup
 
--exec $XTRABACKUP --prepare --target-dir=$targetdir
 
 
 
--echo # Prepare incremental backup
 
--exec $XTRABACKUP --prepare --target-dir=$targetdir --incremental-dir=$incdir > /data/tmp/prepare.log 2>&1
 
 
 
DROP TABLE t;



 Comments   
Comment by Sergei Golubchik [ 2021-10-11 ]

these and similar errors are, of course, nothing new. one can build InnoDB with one set of compression libraries, use mariabackup with a different set of libraries and get the same effect.

although, of course, plugins make it much easier to do inadvertently.

Comment by Elena Stepanova [ 2021-10-11 ]

Certainly. But using mariabackup built on a different system/with a different set of libraries would be such an extreme example of a "corner case" that it had no chance to be even looked at.
On the other hand, with providers it becomes not only standard, but a very likely use case. We do know that people use incremental backup, and we do expect compression users to install providers at some point, not [only] from the very beginning of using the instance. That's all it takes.

Comment by Sergei Golubchik [ 2021-10-11 ] 

commit 810901f8959
 
Author: Sergei Golubchik <serg@mariadb.org>
 
Date:   Mon Oct 11 14:53:11 2021 +0200
 
 
 
    MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup
 
    
    prefer backup-my.cnf from the incremental-dir over the one in target-dir

Comment by Sergei Golubchik [ 2021-10-19 ]

wlad, could you review this please? I'm mainly interested in a second opinion whether this code change can have some unintended consequences.

Comment by Vladislav Vaintroub [ 2021-10-19 ]

Hi serg,
The general idea seems correct, to use the plugins from incremental backup run.

What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?

The test case needs a check for plugin presence, it failed for me on Windows, where non of the compression plugins are present.

I'd suggest renaming encryption_plugin.cc and functions inside it, as it is no more about encryption.

Comment by Sergei Golubchik [ 2021-10-22 ]

What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?

6354
 
	if (prepare && target_dir) {
 
6355
 
		snprintf(conf_file, sizeof(conf_file),
 
6356
 
			 "%s/backup-my.cnf", target_dir);
 
6357
 
			if (!strncmp(argv[1], "--defaults-file=", 16)) {
 
6358
 
				/* Remove defaults-file*/
 
6359
 
				for (int i = 2; ; i++) {
 
6360
 
					if ((argv[i-1]= argv[i]) == 0)
 
6361
 
						break;
 
6362
 
				}
 
6363
 
				argc--;
 
6364
 
			}
 
6365
 
	}

the rest is ok, will do

Comment by Sergei Golubchik [ 2021-10-22 ]

pushed into the preview branch

Generated at Thu Feb 08 09:47:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.