Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31251

MDEV-30968 breaks running mariabackup on older mariadb (opendir(NULL))

    XMLWordPrintable

Details

    Description

      When mariabackup 10.6.13 communicates with an older mariadbd (e.g. 10.6.12), it does not get the aria_log_dir_path from the server. It will stay NULL. This causes an opendir(NULL) down the road.

      Log:

      2023-05-12 08:21:47.129 - DEBUG - [00] 2023-05-12 08:21:47 Finished backing up non-InnoDB tables and files
      2023-05-12 08:21:47.130 - DEBUG - 230512  8:21:47 [ERROR] mysqld got signal 11 ;
      2023-05-12 08:21:47.130 - DEBUG - This could be because you hit a bug. It is also possible that this binary
      2023-05-12 08:21:47.130 - DEBUG - or one of the libraries it was linked against is corrupt, improperly built,
      2023-05-12 08:21:47.130 - DEBUG - or misconfigured. This error can also be caused by malfunctioning hardware.
      2023-05-12 08:21:47.131 - DEBUG - 
      2023-05-12 08:21:47.131 - DEBUG - To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      2023-05-12 08:21:47.131 - DEBUG - 
      2023-05-12 08:21:47.131 - DEBUG - We will try our best to scrape up some info that will hopefully help
      2023-05-12 08:21:47.131 - DEBUG - diagnose the problem, but since we have already crashed,
      2023-05-12 08:21:47.131 - DEBUG - something is definitely wrong and this may fail.
      2023-05-12 08:21:47.131 - DEBUG - 
      2023-05-12 08:21:47.131 - DEBUG - Server version: 10.6.13-MariaDB-1:10.6.13+maria~ubu2004 source revision: a24f2bb50ba4a0dd4127455f7fcdfed584937f36
      2023-05-12 08:21:47.131 - DEBUG - key_buffer_size=0
      2023-05-12 08:21:47.131 - DEBUG - read_buffer_size=131072
      2023-05-12 08:21:47.131 - DEBUG - max_used_connections=0
      2023-05-12 08:21:47.131 - DEBUG - max_threads=1
      2023-05-12 08:21:47.131 - DEBUG - thread_count=0
      2023-05-12 08:21:47.131 - DEBUG - It is possible that mysqld could use up to
      2023-05-12 08:21:47.131 - DEBUG - key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6145 K  bytes of memory
      2023-05-12 08:21:47.131 - DEBUG - Hope that's ok; if not, decrease some variables in the equation.
      2023-05-12 08:21:47.131 - DEBUG - 
      2023-05-12 08:21:47.132 - DEBUG - Thread pointer: 0x0
      2023-05-12 08:21:47.132 - DEBUG - Attempting backtrace. You can use the following information to find out
      2023-05-12 08:21:47.132 - DEBUG - where mysqld died. If you see no messages after this, something went
      2023-05-12 08:21:47.132 - DEBUG - terribly wrong...
      2023-05-12 08:21:47.132 - DEBUG - stack_bottom = 0x0 thread_stack 0x49000
      2023-05-12 08:21:47.134 - DEBUG - Printing to addr2line failed
      2023-05-12 08:21:47.134 - DEBUG - /usr/bin/mariabackup(my_print_stacktrace+0x32)[0x560ffe2e6f82]
      2023-05-12 08:21:47.135 - DEBUG - /usr/bin/mariabackup(handle_fatal_signal+0x485)[0x560ffde1c025]
      2023-05-12 08:21:47.136 - DEBUG - /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f0c4f0d6420]
      2023-05-12 08:21:47.138 - DEBUG - /lib/x86_64-linux-gnu/libc.so.6(opendir+0x24)[0x7f0c4ec74ea4]
      2023-05-12 08:21:47.140 - DEBUG - /usr/bin/mariabackup(+0x7101e4)[0x560ffda771e4]
      2023-05-12 08:21:47.140 - DEBUG - /usr/bin/mariabackup(_Z12backup_startP7ds_ctxtS0_R14CorruptedPages+0x10b)[0x560ffda7a9bb]
      2023-05-12 08:21:47.141 - DEBUG - /usr/bin/mariabackup(+0x6fbc04)[0x560ffda62c04]
      2023-05-12 08:21:47.141 - DEBUG - /usr/bin/mariabackup(main+0x182)[0x560ffda0a122]
      2023-05-12 08:21:47.143 - DEBUG - /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f0c4ebbb083]
      2023-05-12 08:21:47.144 - DEBUG - /usr/bin/mariabackup(_start+0x2e)[0x560ffda4ea1e]
      

      The problem is caused by:
      https://github.com/MariaDB/server/commit/9f98a2acd71dcfbdb32a08e72a4737359ed9be40

      Specifically this:
      https://github.com/MariaDB/server/commit/9f98a2acd71dcfbdb32a08e72a4737359ed9be40#diff-a1b3373388d92a9bf267ea6336564fe8da01b98d1d8c59ed3758c252872a632eR1506-R1508

      There we do:

              if (!backup_files_from_datadir(ds_data, fil_path_to_mysql_datadir,
      	                               "aws-kms-key") ||
                  !backup_files_from_datadir(ds_data,
                                             aria_log_dir_path,
                                             "aria_log")) {
      		return false;
      	}
      

      But if aria_log_dir_path is NULL, we go down to:
      https://github.com/MariaDB/server/blob/0474466bc2c3e05a160677f1dbabea374e942736/extra/mariabackup/backup_copy.cc#L2270-L2274

      Calling opendir(NULL), causing a segfault in libc.

      Possible fix:

      diff --git a/extra/mariabackup/backup_copy.cc b/extra/mariabackup/backup_copy.cc
      index dbf12cedd68..33ebf55f45f 100644
      --- a/extra/mariabackup/backup_copy.cc
      +++ b/extra/mariabackup/backup_copy.cc
      @@ -1439,9 +1439,9 @@ bool backup_start(ds_ctxt *ds_data, ds_ctxt *ds_meta,
       
               if (!backup_files_from_datadir(ds_data, fil_path_to_mysql_datadir,
                                             "aws-kms-key") ||
      -            !backup_files_from_datadir(ds_data,
      +            (aria_log_dir_path && !backup_files_from_datadir(ds_data,
                                              aria_log_dir_path,
      -                                       "aria_log")) {
      +                                       "aria_log"))) {
                      return false;
              }
       
      

      Or, a workaround for callers:

      --- a/mariadb-backup.py
      +++ b/mariadb-backup.py
      @@ -43,6 +43,7 @@ MARIABACKUP_THREADS = (MARIABACKUP_THREADS, 1)[MARIABACKUP_THREADS < 1]
       MARIABACKUP_BASEARGS = (
           '--defaults-file={}'.format(CNFFILE),
           '--backup',
      +    '--aria-log-dir-path=.',  # see: MDEV-31251
           '--parallel={}'.format(MARIABACKUP_THREADS))
       
       INSTANCE_HASH = '<gets_set_in_main>'
      

      Cheers,
      Walter Doekes
      OSSO B.V.

      Attachments

        Issue Links

          Activity

            People

              bar Alexander Barkov
              wdoekes Walter Doekes
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.