Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15780

Mariabackup fails on --prepare stage when innodb_data_file_path contains a path

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1(EOL), 10.2(EOL), 10.3(EOL)
    • 10.1.33, 10.2.15
    • Backup
    • None

    Description

      If you set innodb_data_file_path to include a path, then the --prepare step fails when using mariabackup.

      For instance, suppose you have the following 2 items set:

      datadir= /var/lib/mysql
      innodb_data_file_path=/var/lib/mysql/ibdata1:12M:autoextend
      

      Now, if you run mariabackup with --backup, it will run fine.

      However, when you attempt mariabackup with --prepare, it fails with:

      chris@chris-linux-laptop-64:~/Desktop$ sudo mariabackup --prepare --target-dir=/mysqlbackup 
      mariabackup based on MariaDB server 10.2.14-MariaDB debian-linux-gnu (x86_64) 
      mariabackup: cd to /mysqlbackup/
      mariabackup: This target seems to be not prepared yet.
      mariabackup: using the following InnoDB configuration for recovery:
      mariabackup:   innodb_data_home_dir = .
      mariabackup:   innodb_data_file_path = /var/lib/mysql/ibdata1:12M:autoextend
      mariabackup:   innodb_log_group_home_dir = .
      mariabackup: Starting InnoDB instance for recovery.
      mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Uses event mutexes
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Compressed tables use zlib 1.2.8
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Number of pools: 1
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Using SSE2 crc32 instructions
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Completed initialization of buffer pool
      2018-04-04 14:38:39 139954027566848 [Note] InnoDB: page_cleaner coordinator priority: -20
      2018-04-04 14:38:39 139954270910336 [Note] InnoDB: The first innodb_system data file '/var/lib/mysql/ibdata1' did not exist. A new tablespace will be created!
      2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: Operating system error number 2 in a file operation.
      2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: The error means the system cannot find the path specified.
      2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
      2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: File .//var/lib/mysql/ibdata1: 'create' returned OS error 71. Cannot continue operation
      180404 14:38:39 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.14-MariaDB-10.2.14+maria~trusty
      key_buffer_size=0
      read_buffer_size=131072
      max_used_connections=0
      max_threads=1
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      addr2line: 'mariabackup': No such file
      mariabackup(my_print_stacktrace+0x2e)[0x7f49a55e4a4e]
      mariabackup(handle_fatal_signal+0x355)[0x7f49a50d3175]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f49a4599340]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f49a2950f79]
      linux/raise.c:56(__GI_raise)[0x7f49a2954388]
      addr2line: 'mariabackup': No such file
      mariabackup(+0x9abe25)[0x7f49a5377e25]
      mariabackup(+0x8f8735)[0x7f49a52c4735]
      mariabackup(+0x8fdf9a)[0x7f49a52c9f9a]
      mariabackup(+0xa4a97c)[0x7f49a541697c]
      mariabackup(+0x458252)[0x7f49a4e24252]
      mariabackup(main+0x185)[0x7f49a4e02b95]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f49a293bec5]
      addr2line: 'mariabackup': No such file
      mariabackup(+0x4501fd)[0x7f49a4e1c1fd]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      

      If you just have innodb_data_file_path set to:

      innodb_data_file_path=ibdata1:12M:autoextend
      

      Then it works as expected.

      Alternatively, if you edit the backup-my.cnf file that is generated in the --target-dir and remove the path from innodb_data_file_path, then it also works as expected.

      This extra step is not needed in Xtrabackup.

      Attachments

        Activity

          ccalender, with absolute paths in innodb_data_file_path, I cannot get mysqld to start. this is what it tells me,on Linux

          2018-04-11  9:05:25 0 [ERROR] InnoDB: File .//home/wlad/ibdata1: 'create' returned OS error 71. Cannot continue operatio
          n
          180411  9:05:25 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.
          
          

          Windows is similar. To get a backup , one would first need a server started with such parameter, and that does not quite work. I'm not sure how it is supposed to work. Any idea?

          wlad Vladislav Vaintroub added a comment - ccalender , with absolute paths in innodb_data_file_path, I cannot get mysqld to start. this is what it tells me,on Linux 2018-04-11 9:05:25 0 [ERROR] InnoDB: File .//home/wlad/ibdata1: 'create' returned OS error 71. Cannot continue operatio n 180411 9:05:25 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Windows is similar. To get a backup , one would first need a server started with such parameter, and that does not quite work. I'm not sure how it is supposed to work. Any idea?

          Hi wlad,

          There is a way to get this to work on Linux. I initially tried with Windows, since that is what I had most handy, and encountered the same issue as you above. However, when I switched to Linux (Ubuntu, specifically), I was able to get mysqld started and then run mariabackup (as you can see from the results). This is also what the other reporter did (and why they informed me about the bug).

          To keep things simple, I first started mysqld with the following:

          datadir= /var/lib/mysql
          innodb_data_file_path=ibdata1:12M:autoextend
          

          Once that worked, I then modified innodb_data_file_path to:

          datadir= /var/lib/mysql
          innodb_data_file_path=/var/lib/mysql/ibdata1:12M:autoextend
          

          I did not have any other path settings in effect, such as innodb_data_home_dir and innodb_log_group_home_dir.

          I did some more testing last night, and will be filing a separate bug today about how mysqld will not even start on Windows if you give it a full path starting with "C:..". Perhaps the two are involved, but again, both the reporter and I were able to get mysqld started with these values on Linux, and thus able to produce the mariabackup bug I reported here.

          ccalender Chris Calender (Inactive) added a comment - Hi wlad , There is a way to get this to work on Linux. I initially tried with Windows, since that is what I had most handy, and encountered the same issue as you above. However, when I switched to Linux (Ubuntu, specifically), I was able to get mysqld started and then run mariabackup (as you can see from the results). This is also what the other reporter did (and why they informed me about the bug). To keep things simple, I first started mysqld with the following: datadir= /var/lib/mysql innodb_data_file_path=ibdata1:12M:autoextend Once that worked, I then modified innodb_data_file_path to: datadir= /var/lib/mysql innodb_data_file_path=/var/lib/mysql/ibdata1:12M:autoextend I did not have any other path settings in effect, such as innodb_data_home_dir and innodb_log_group_home_dir. I did some more testing last night, and will be filing a separate bug today about how mysqld will not even start on Windows if you give it a full path starting with "C:..". Perhaps the two are involved, but again, both the reporter and I were able to get mysqld started with these values on Linux, and thus able to produce the mariabackup bug I reported here.
          wlad Vladislav Vaintroub added a comment - - edited

          my attempt was from Linux (.//home/wlad/ibdata1 shows it), ubuntu specifically, after I failed to get it working from Windows prior to that.

          I set datadir = to my $HOME (which is /home/wlad),
          What do I do wrong?
          marko says btw, full paths are not supposed to work. With innodb, and by extension, with backup

          wlad Vladislav Vaintroub added a comment - - edited my attempt was from Linux (.//home/wlad/ibdata1 shows it), ubuntu specifically, after I failed to get it working from Windows prior to that. I set datadir = to my $HOME (which is /home/wlad), What do I do wrong? marko says btw, full paths are not supposed to work. With innodb, and by extension, with backup
          wlad Vladislav Vaintroub added a comment - - edited

          Does not quite work.

          if I do

          sql/mysqld --no-defaults --lc-messages-dir=/home/mariadb/wlad/1 0.2/xxx/sql/share --datadir=$HOME --innodb-data-file-path=$HOME/ibdata1:10M:auto extend --skip-grant-tables

          I get

          2018-04-11 18:10:51 140303482238784 [Note] sql/mysqld (mysqld 10.2.15-MariaDB-debug) starting as process 11128 ...
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Uses event mutexes
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Compressed tables use zlib 1.2.8
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using Linux native AIO
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Number of pools: 1
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using SSE2 crc32 instructions
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Completed initialization of buffer pool
          2018-04-11 18:10:51 140302758807296 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
          2018-04-11 18:10:51 140303482238784 [Note] InnoDB: The first innodb_system data file '/home/mariadb/ibdata1' did not exist. A new tablespace will be created!
          2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: Operating system error number 2 in a file operation.
          2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: The error means the system cannot find the path specified.
          2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
          2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: File .//home/mariadb/ibdata1: 'create' returned OS error 71. Cannot continue operation
          180411 18:10:51 [ERROR] mysqld got signal 6 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.
           
          To report this bug, see https://mariadb.com/kb/en/reporting-bugs
           
          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed,
          something is definitely wrong and this may fail.
           
          Server version: 10.2.15-MariaDB-debug
          key_buffer_size=134217728
          read_buffer_size=131072
          max_used_connections=0
          max_threads=153
          thread_count=0
          It is possible that mysqld could use up to
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467349 K  bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.
           
          Thread pointer: 0x0
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          

          wlad Vladislav Vaintroub added a comment - - edited Does not quite work. if I do sql/mysqld --no-defaults --lc-messages-dir=/home/mariadb/wlad/1 0.2/xxx/sql/share --datadir=$HOME --innodb-data-file-path=$HOME/ibdata1:10M:auto extend --skip-grant-tables I get 2018-04-11 18:10:51 140303482238784 [Note] sql/mysqld (mysqld 10.2.15-MariaDB-debug) starting as process 11128 ... 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!! 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Uses event mutexes 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Compressed tables use zlib 1.2.8 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using Linux native AIO 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Number of pools: 1 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using SSE2 crc32 instructions 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Completed initialization of buffer pool 2018-04-11 18:10:51 140302758807296 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2018-04-11 18:10:51 140303482238784 [Note] InnoDB: The first innodb_system data file '/home/mariadb/ibdata1' did not exist. A new tablespace will be created! 2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: Operating system error number 2 in a file operation. 2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: The error means the system cannot find the path specified. 2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them. 2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: File .//home/mariadb/ibdata1: 'create' returned OS error 71. Cannot continue operation 180411 18:10:51 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.2.15-MariaDB-debug key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467349 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...

          Ok, got it empty innodb-data-home-dir is necessary as well.

          wlad Vladislav Vaintroub added a comment - Ok, got it empty innodb-data-home-dir is necessary as well.

          People

            wlad Vladislav Vaintroub
            ccalender Chris Calender (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.