It looks like upon prepare of an incremental backup MariaBackup reads options (at least plugins) from the config file stored in the original full backup directory. So, if a provider was added since the original backup, MariaBackup cannot prepare tables compressed with the added algorithms.
The MTR test case below simply hangs upon the 2nd prepare. If the output is redirected to a log, the last lines after which mariabackup hangs are
2021-10-10 0:15:44 0 [Note] InnoDB: Starting final batch to recover 6 pages from redo log.
2021-10-10 0:15:44 0 [Warning] InnoDB: 'test/t_snappy.ibd' is compressed with snappy, which is not currently loaded
and the stack trace is
Thread 1 (Thread 0x7ff4e0610800 (LWP 3598619)):
#0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5605dab74cf8 <recv_sys+216>) at ../sysdeps/nptl/futex-internal.h:183
#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5605dab74c48 <recv_sys+40>, cond=0x5605dab74cd0 <recv_sys+176>) at pthread_cond_wait.c:508
#2 __pthread_cond_wait (cond=0x5605dab74cd0 <recv_sys+176>, mutex=0x5605dab74c48 <recv_sys+40>) at pthread_cond_wait.c:638
#3 0x00005605da03def2 in safe_cond_wait (cond=0x5605dab74cd0 <recv_sys+176>, mp=0x5605dab74c20 <recv_sys>, file=0x5605da4f1ba0 "/data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/log/log0recv.cc", line=3143) at /data/src/preview-10.7-MDEV-12933-provider-plugins/mysys/thr_mutex.c:492
#4 0x00005605d9d4ce7d in recv_sys_t::apply (this=0x5605dab74c20 <recv_sys>, last_batch=true) at /data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/log/log0recv.cc:3143
#5 0x00005605d9e8e001 in srv_start (create_new_db=false) at /data/src/preview-10.7-MDEV-12933-provider-plugins/storage/innobase/srv/srv0start.cc:1488
#6 0x00005605d9216e06 in innodb_init () at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:2144
#7 0x00005605d922290c in xtrabackup_prepare_func (argv=0x5605dba614c8) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:5891
#8 0x00005605d92252f8 in main_low (argv=0x5605dba614c8) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:6793
#9 0x00005605d9224b29 in main (argc=4, argv=0x7ffd3c3ff558) at /data/src/preview-10.7-MDEV-12933-provider-plugins/extra/mariabackup/xtrabackup.cc:6591
these and similar errors are, of course, nothing new. one can build InnoDB with one set of compression libraries, use mariabackup with a different set of libraries and get the same effect.
although, of course, plugins make it much easier to do inadvertently.
Sergei Golubchik
added a comment - these and similar errors are, of course, nothing new. one can build InnoDB with one set of compression libraries, use mariabackup with a different set of libraries and get the same effect.
although, of course, plugins make it much easier to do inadvertently.
Certainly. But using mariabackup built on a different system/with a different set of libraries would be such an extreme example of a "corner case" that it had no chance to be even looked at.
On the other hand, with providers it becomes not only standard, but a very likely use case. We do know that people use incremental backup, and we do expect compression users to install providers at some point, not [only] from the very beginning of using the instance. That's all it takes.
Elena Stepanova
added a comment - Certainly. But using mariabackup built on a different system/with a different set of libraries would be such an extreme example of a "corner case" that it had no chance to be even looked at.
On the other hand, with providers it becomes not only standard, but a very likely use case. We do know that people use incremental backup, and we do expect compression users to install providers at some point, not [only] from the very beginning of using the instance. That's all it takes.
MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup
prefer backup-my.cnf from the incremental-dir over the one in target-dir
Sergei Golubchik
added a comment -
commit 810901f8959
Author: Sergei Golubchik <serg@mariadb.org>
Date: Mon Oct 11 14:53:11 2021 +0200
MDEV-26794 MariaBackup does not recognize added providers upon prepare of incremental backup
prefer backup-my.cnf from the incremental-dir over the one in target-dir
wlad, could you review this please? I'm mainly interested in a second opinion whether this code change can have some unintended consequences.
Sergei Golubchik
added a comment - wlad , could you review this please? I'm mainly interested in a second opinion whether this code change can have some unintended consequences.
Hi serg,
The general idea seems correct, to use the plugins from incremental backup run.
What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?
The test case needs a check for plugin presence, it failed for me on Windows, where non of the compression plugins are present.
I'd suggest renaming encryption_plugin.cc and functions inside it, as it is no more about encryption.
Vladislav Vaintroub
added a comment - Hi serg ,
The general idea seems correct, to use the plugins from incremental backup run.
What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?
The test case needs a check for plugin presence, it failed for me on Windows, where non of the compression plugins are present.
I'd suggest renaming encryption_plugin.cc and functions inside it, as it is no more about encryption.
What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?
6354
if (prepare && target_dir) {
6355
snprintf(conf_file, sizeof(conf_file),
6356
"%s/backup-my.cnf", target_dir);
6357
if (!strncmp(argv[1], "--defaults-file=", 16)) {
6358
/* Remove defaults-file*/
6359
for (int i = 2; ; i++) {
6360
if ((argv[i-1]= argv[i]) == 0)
6361
break;
6362
}
6363
argc--;
6364
}
6365
}
the rest is ok, will do
Sergei Golubchik
added a comment - What I did not understand, is setting target_dir to incremental_dir in xtrabackup.cc. Why is it necessary?
6354
if (prepare && target_dir) {
6355
snprintf(conf_file, sizeof (conf_file),
6356
"%s/backup-my.cnf" , target_dir);
6357
if (! strncmp (argv[1], "--defaults-file=" , 16)) {
6358
/* Remove defaults-file*/
6359
for ( int i = 2; ; i++) {
6360
if ((argv[i-1]= argv[i]) == 0)
6361
break ;
6362
}
6363
argc--;
6364
}
6365
}
the rest is ok, will do
these and similar errors are, of course, nothing new. one can build InnoDB with one set of compression libraries, use mariabackup with a different set of libraries and get the same effect.
although, of course, plugins make it much easier to do inadvertently.