[MDEV-31251] MDEV-30968 breaks running mariabackup on older mariadb (opendir(NULL)) Created: 2023-05-12  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.4.29, 10.6.13
Fix Version/s: 10.4, 10.5, 10.6, 10.11, 11.0

Type: Bug Priority: Major
Reporter: Walter Doekes Assignee: Alexander Barkov
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Problem/Incident
is caused by MDEV-30968 mariadb-backup does not copy Aria log... Closed

 Description   

When mariabackup 10.6.13 communicates with an older mariadbd (e.g. 10.6.12), it does not get the aria_log_dir_path from the server. It will stay NULL. This causes an opendir(NULL) down the road.

Log:

2023-05-12 08:21:47.129 - DEBUG - [00] 2023-05-12 08:21:47 Finished backing up non-InnoDB tables and files
 
2023-05-12 08:21:47.130 - DEBUG - 230512  8:21:47 [ERROR] mysqld got signal 11 ;
 
2023-05-12 08:21:47.130 - DEBUG - This could be because you hit a bug. It is also possible that this binary
 
2023-05-12 08:21:47.130 - DEBUG - or one of the libraries it was linked against is corrupt, improperly built,
 
2023-05-12 08:21:47.130 - DEBUG - or misconfigured. This error can also be caused by malfunctioning hardware.
 
2023-05-12 08:21:47.131 - DEBUG - 
2023-05-12 08:21:47.131 - DEBUG - To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
2023-05-12 08:21:47.131 - DEBUG - 
2023-05-12 08:21:47.131 - DEBUG - We will try our best to scrape up some info that will hopefully help
 
2023-05-12 08:21:47.131 - DEBUG - diagnose the problem, but since we have already crashed,
 
2023-05-12 08:21:47.131 - DEBUG - something is definitely wrong and this may fail.
 
2023-05-12 08:21:47.131 - DEBUG - 
2023-05-12 08:21:47.131 - DEBUG - Server version: 10.6.13-MariaDB-1:10.6.13+maria~ubu2004 source revision: a24f2bb50ba4a0dd4127455f7fcdfed584937f36
 
2023-05-12 08:21:47.131 - DEBUG - key_buffer_size=0
 
2023-05-12 08:21:47.131 - DEBUG - read_buffer_size=131072
 
2023-05-12 08:21:47.131 - DEBUG - max_used_connections=0
 
2023-05-12 08:21:47.131 - DEBUG - max_threads=1
 
2023-05-12 08:21:47.131 - DEBUG - thread_count=0
 
2023-05-12 08:21:47.131 - DEBUG - It is possible that mysqld could use up to
 
2023-05-12 08:21:47.131 - DEBUG - key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6145 K  bytes of memory
 
2023-05-12 08:21:47.131 - DEBUG - Hope that's ok; if not, decrease some variables in the equation.
 
2023-05-12 08:21:47.131 - DEBUG - 
2023-05-12 08:21:47.132 - DEBUG - Thread pointer: 0x0
 
2023-05-12 08:21:47.132 - DEBUG - Attempting backtrace. You can use the following information to find out
 
2023-05-12 08:21:47.132 - DEBUG - where mysqld died. If you see no messages after this, something went
 
2023-05-12 08:21:47.132 - DEBUG - terribly wrong...
 
2023-05-12 08:21:47.132 - DEBUG - stack_bottom = 0x0 thread_stack 0x49000
 
2023-05-12 08:21:47.134 - DEBUG - Printing to addr2line failed
 
2023-05-12 08:21:47.134 - DEBUG - /usr/bin/mariabackup(my_print_stacktrace+0x32)[0x560ffe2e6f82]
 
2023-05-12 08:21:47.135 - DEBUG - /usr/bin/mariabackup(handle_fatal_signal+0x485)[0x560ffde1c025]
 
2023-05-12 08:21:47.136 - DEBUG - /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f0c4f0d6420]
 
2023-05-12 08:21:47.138 - DEBUG - /lib/x86_64-linux-gnu/libc.so.6(opendir+0x24)[0x7f0c4ec74ea4]
 
2023-05-12 08:21:47.140 - DEBUG - /usr/bin/mariabackup(+0x7101e4)[0x560ffda771e4]
 
2023-05-12 08:21:47.140 - DEBUG - /usr/bin/mariabackup(_Z12backup_startP7ds_ctxtS0_R14CorruptedPages+0x10b)[0x560ffda7a9bb]
 
2023-05-12 08:21:47.141 - DEBUG - /usr/bin/mariabackup(+0x6fbc04)[0x560ffda62c04]
 
2023-05-12 08:21:47.141 - DEBUG - /usr/bin/mariabackup(main+0x182)[0x560ffda0a122]
 
2023-05-12 08:21:47.143 - DEBUG - /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f0c4ebbb083]
 
2023-05-12 08:21:47.144 - DEBUG - /usr/bin/mariabackup(_start+0x2e)[0x560ffda4ea1e]

The problem is caused by:
https://github.com/MariaDB/server/commit/9f98a2acd71dcfbdb32a08e72a4737359ed9be40

Specifically this:
https://github.com/MariaDB/server/commit/9f98a2acd71dcfbdb32a08e72a4737359ed9be40#diff-a1b3373388d92a9bf267ea6336564fe8da01b98d1d8c59ed3758c252872a632eR1506-R1508

There we do:

        if (!backup_files_from_datadir(ds_data, fil_path_to_mysql_datadir,
 
	                               "aws-kms-key") ||
 
            !backup_files_from_datadir(ds_data,
 
                                       aria_log_dir_path,
 
                                       "aria_log")) {
 
		return false;
 
	}

But if aria_log_dir_path is NULL, we go down to:
https://github.com/MariaDB/server/blob/0474466bc2c3e05a160677f1dbabea374e942736/extra/mariabackup/backup_copy.cc#L2270-L2274

Calling opendir(NULL), causing a segfault in libc.

Possible fix:

diff --git a/extra/mariabackup/backup_copy.cc b/extra/mariabackup/backup_copy.cc
 
index dbf12cedd68..33ebf55f45f 100644
 
--- a/extra/mariabackup/backup_copy.cc
 
+++ b/extra/mariabackup/backup_copy.cc
 
@@ -1439,9 +1439,9 @@ bool backup_start(ds_ctxt *ds_data, ds_ctxt *ds_meta,
 
 
         if (!backup_files_from_datadir(ds_data, fil_path_to_mysql_datadir,
 
                                       "aws-kms-key") ||
 
-            !backup_files_from_datadir(ds_data,
 
+            (aria_log_dir_path && !backup_files_from_datadir(ds_data,
 
                                        aria_log_dir_path,
 
-                                       "aria_log")) {
 
+                                       "aria_log"))) {
 
                return false;
 
        }

Or, a workaround for callers:

--- a/mariadb-backup.py
 
+++ b/mariadb-backup.py
 
@@ -43,6 +43,7 @@ MARIABACKUP_THREADS = (MARIABACKUP_THREADS, 1)[MARIABACKUP_THREADS < 1]
 
 MARIABACKUP_BASEARGS = (
 
     '--defaults-file={}'.format(CNFFILE),
 
     '--backup',
 
+    '--aria-log-dir-path=.',  # see: MDEV-31251
 
     '--parallel={}'.format(MARIABACKUP_THREADS))
 
 
 INSTANCE_HASH = '<gets_set_in_main>'

Cheers,
Walter Doekes
OSSO B.V.
Generated at Thu Feb 08 10:22:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.