Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12988

backup fails if innodb_undo_tablespaces>0

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.24
    • 10.1.27, 10.2.9, 10.3.2
    • Backup
    • None
    • CentOS 7.x x86_64

    Description

      # mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
      170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
      Using server version 10.1.24-MariaDB
      mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64) 
      xtrabackup: uses posix_fadvise().
      xtrabackup: cd to /opt/mysql
      xtrabackup: open files limit requested 65535, set to 100001
      xtrabackup: using the following InnoDB configuration:
      xtrabackup:   innodb_data_home_dir = /opt/mysql
      xtrabackup:   innodb_data_file_path = ibdata1:128M:autoextend
      xtrabackup:   innodb_log_group_home_dir = /opt/mysql
      xtrabackup:   innodb_log_files_in_group = 2
      xtrabackup:   innodb_log_file_size = 268435456
      2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
      xtrabackup: using O_DIRECT
      xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
      170603 08:21:55 >> log scanned up to (376182854)
      

      2017-06-03 08:21:55 7f1f34a25900  InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
      InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      170603  8:21:55 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.1.24-MariaDB
      key_buffer_size=0
      read_buffer_size=131072
      max_used_connections=0
      max_threads=1
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x48400
      addr2line: 'mariabackup': No such file
      mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
      mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
      /lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
      /lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
      /lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
      mariabackup(+0x91862b)[0x7f1f3538362b]
      mariabackup(+0x3a7de4)[0x7f1f34e12de4]
      mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
      mariabackup(main+0xb4d)[0x7f1f34df899d]
      /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
      mariabackup(+0x3a624d)[0x7f1f34e1124d]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      Aborted
      

      Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.

      Attachments

        Issue Links

          Activity

            mxu Michael Xu created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Description # mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
            170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
            Using server version 10.1.24-MariaDB
            mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64)
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /opt/mysql
            xtrabackup: open files limit requested 65535, set to 100001
            xtrabackup: using the following InnoDB configuration:
            xtrabackup: innodb_data_home_dir = /opt/mysql
            xtrabackup: innodb_data_file_path = ibdata1:128M:autoextend
            xtrabackup: innodb_log_group_home_dir = /opt/mysql
            xtrabackup: innodb_log_files_in_group = 2
            xtrabackup: innodb_log_file_size = 268435456
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
            xtrabackup: using O_DIRECT
            xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
            170603 08:21:55 >> log scanned up to (376182854)
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
            InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            170603 8:21:55 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.1.24-MariaDB
            key_buffer_size=0
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x48400
            addr2line: 'mariabackup': No such file
            mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
            mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
            /lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
            /lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
            /lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
            mariabackup(+0x91862b)[0x7f1f3538362b]
            mariabackup(+0x3a7de4)[0x7f1f34e12de4]
            mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
            mariabackup(main+0xb4d)[0x7f1f34df899d]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
            mariabackup(+0x3a624d)[0x7f1f34e1124d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Aborted


            Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.
            {noformat}
            # mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
            170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
            Using server version 10.1.24-MariaDB
            mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64)
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /opt/mysql
            xtrabackup: open files limit requested 65535, set to 100001
            xtrabackup: using the following InnoDB configuration:
            xtrabackup: innodb_data_home_dir = /opt/mysql
            xtrabackup: innodb_data_file_path = ibdata1:128M:autoextend
            xtrabackup: innodb_log_group_home_dir = /opt/mysql
            xtrabackup: innodb_log_files_in_group = 2
            xtrabackup: innodb_log_file_size = 268435456
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
            xtrabackup: using O_DIRECT
            xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
            170603 08:21:55 >> log scanned up to (376182854)
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
            InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            170603 8:21:55 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.1.24-MariaDB
            key_buffer_size=0
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x48400
            addr2line: 'mariabackup': No such file
            mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
            mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
            /lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
            /lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
            /lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
            mariabackup(+0x91862b)[0x7f1f3538362b]
            mariabackup(+0x3a7de4)[0x7f1f34e12de4]
            mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
            mariabackup(main+0xb4d)[0x7f1f34df899d]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
            mariabackup(+0x3a624d)[0x7f1f34e1124d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Aborted
            {noformat}

            Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.
            elenst Elena Stepanova made changes -
            Description {noformat}
            # mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
            170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
            Using server version 10.1.24-MariaDB
            mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64)
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /opt/mysql
            xtrabackup: open files limit requested 65535, set to 100001
            xtrabackup: using the following InnoDB configuration:
            xtrabackup: innodb_data_home_dir = /opt/mysql
            xtrabackup: innodb_data_file_path = ibdata1:128M:autoextend
            xtrabackup: innodb_log_group_home_dir = /opt/mysql
            xtrabackup: innodb_log_files_in_group = 2
            xtrabackup: innodb_log_file_size = 268435456
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
            xtrabackup: using O_DIRECT
            xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
            170603 08:21:55 >> log scanned up to (376182854)
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
            InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            170603 8:21:55 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.1.24-MariaDB
            key_buffer_size=0
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x48400
            addr2line: 'mariabackup': No such file
            mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
            mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
            /lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
            /lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
            /lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
            mariabackup(+0x91862b)[0x7f1f3538362b]
            mariabackup(+0x3a7de4)[0x7f1f34e12de4]
            mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
            mariabackup(main+0xb4d)[0x7f1f34df899d]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
            mariabackup(+0x3a624d)[0x7f1f34e1124d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Aborted
            {noformat}

            Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.
            {noformat}
            # mariabackup --defaults-file=/etc/my.cnf --backup --compress --host=127.0.0.1 --port=3306 --user=backup --password=changeme --compress-threads=16 --parallel=16 --extra-lsndir=/var/local-backup --stream=xbstream --tmpdir=/tmp/backup_3306 > /opt/backup/databases/backup_2017_06_03_08_08.xbs
            170603 08:21:55 Connecting to MySQL server host: 127.0.0.1, user: backup, password: set, port: 3306, socket: not set
            Using server version 10.1.24-MariaDB
            mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64)
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /opt/mysql
            xtrabackup: open files limit requested 65535, set to 100001
            xtrabackup: using the following InnoDB configuration:
            xtrabackup: innodb_data_home_dir = /opt/mysql
            xtrabackup: innodb_data_file_path = ibdata1:128M:autoextend
            xtrabackup: innodb_log_group_home_dir = /opt/mysql
            xtrabackup: innodb_log_files_in_group = 2
            xtrabackup: innodb_log_file_size = 268435456
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Using Linux native AIO
            xtrabackup: using O_DIRECT
            xtrabackupp: Warning: ignoring innodb_flush_method = O_DIRECT on Windows.
            170603 08:21:55 >> log scanned up to (376182854)
            {noformat}
            {noformat}
            2017-06-03 08:21:55 7f1f34a25900 InnoDB: Assertion failure in thread 139772003768576 in file srv0start.cc line 1475
            InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
            InnoDB: We intentionally generate a memory trap.
            InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
            InnoDB: If you get repeated assertion failures or crashes, even
            InnoDB: immediately after the mysqld startup, there may be
            InnoDB: corruption in the InnoDB tablespace. Please refer to
            InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
            InnoDB: about forcing recovery.
            170603 8:21:55 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.1.24-MariaDB
            key_buffer_size=0
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5301 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x48400
            addr2line: 'mariabackup': No such file
            mariabackup(my_print_stacktrace+0x2e)[0x7f1f355325be]
            mariabackup(handle_fatal_signal+0x305)[0x7f1f3505c685]
            /lib64/libpthread.so.0(+0xf370)[0x7f1f3463c370]
            /lib64/libc.so.6(gsignal+0x37)[0x7f1f329911d7]
            /lib64/libc.so.6(abort+0x148)[0x7f1f329928c8]
            mariabackup(+0x91862b)[0x7f1f3538362b]
            mariabackup(+0x3a7de4)[0x7f1f34e12de4]
            mariabackup(_Z22xtrabackup_backup_funcv+0xc6e)[0x7f1f34e1aa6e]
            mariabackup(main+0xb4d)[0x7f1f34df899d]
            /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1f3297db35]
            mariabackup(+0x3a624d)[0x7f1f34e1124d]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Aborted
            {noformat}

            Also, there is a warning, ignoring innodb_flush_method = O_DIRECT on Windows, but the server is running on Linux.
            elenst Elena Stepanova made changes -
            Fix Version/s 10.1 [ 16100 ]
            Assignee Vladislav Vaintroub [ wlad ]

            Please share your my.cnf file please.

            wlad Vladislav Vaintroub added a comment - Please share your my.cnf file please.
            wlad Vladislav Vaintroub made changes -
            Priority Critical [ 2 ] Major [ 3 ]

            I can repeat the problem with 10.1.24 and latest 10.1 build by calling mysql_install_db with innodb_undo_tablespaces=4 .

            10.1.24/bin/mariabackup --defaults-file=my.cnf --target-dir=bkup --backup
            170614 17:44:06 Connecting to MySQL server host: localhost, user: root, password: not set, port: 3313, socket: /home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
            Using server version 10.1.24-MariaDB
            ../_depot/m-tar/10.1.24/bin/mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64) 
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /home/a/test1/mariadb-environs/m7-10.1.24/dt
            xtrabackup: open files limit requested 0, set to 1024
            xtrabackup: using the following InnoDB configuration:
            xtrabackup:   innodb_data_home_dir = ./
            xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
            xtrabackup:   innodb_log_group_home_dir = ./
            xtrabackup:   innodb_log_files_in_group = 2
            xtrabackup:   innodb_log_file_size = 50331648
            170614 17:44:06 >> log scanned up to (1501756)
            2017-06-14 17:44:06 7f622943b640  InnoDB: Assertion failure in thread 140059575825984 in file srv0start.cc line 1475
            InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
            

            my.cnf

            [xtrabackup]
            user=root
            port=3313
             
            [client]
            user=root
            port=3313
            socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
             
            [mysqld]
            server_id=7
            port=3313
            socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock
            datadir=/home/a/test1/mariadb-environs/m7-10.1.24/dt
            log-error=/home/a/test1/mariadb-environs/m7-10.1.24/dt/error.log
             
            pid_file=/home/a/test1/mariadb-environs/m7-10.1.24/dt/p.id
             
            lc_messages_dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/share
            plugin-dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/lib/plugin
            innodb_undo_tablespaces=4
            

            10.1.23 doesn't seem to be affected.

            anikitin Andrii Nikitin (Inactive) added a comment - I can repeat the problem with 10.1.24 and latest 10.1 build by calling mysql_install_db with innodb_undo_tablespaces=4 . 10.1.24/bin/mariabackup --defaults-file=my.cnf --target-dir=bkup --backup 170614 17:44:06 Connecting to MySQL server host: localhost, user: root, password: not set, port: 3313, socket: /home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock Using server version 10.1.24-MariaDB ../_depot/m-tar/10.1.24/bin/mariabackup based on MariaDB server 10.1.24-MariaDB Linux (x86_64) xtrabackup: uses posix_fadvise(). xtrabackup: cd to /home/a/test1/mariadb-environs/m7-10.1.24/dt xtrabackup: open files limit requested 0, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 170614 17:44:06 >> log scanned up to (1501756) 2017-06-14 17:44:06 7f622943b640 InnoDB: Assertion failure in thread 140059575825984 in file srv0start.cc line 1475 InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i] my.cnf [xtrabackup] user=root port=3313   [client] user=root port=3313 socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock   [mysqld] server_id=7 port=3313 socket=/home/a/test1/mariadb-environs/m7-10.1.24/dt/my.sock datadir=/home/a/test1/mariadb-environs/m7-10.1.24/dt log-error=/home/a/test1/mariadb-environs/m7-10.1.24/dt/error.log   pid_file=/home/a/test1/mariadb-environs/m7-10.1.24/dt/p.id   lc_messages_dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/share plugin-dir=/home/a/test1/mariadb-environs/m7-10.1.24/../_depot/m-tar/10.1.24/lib/plugin innodb_undo_tablespaces=4 10.1.23 doesn't seem to be affected.
            anikitin Andrii Nikitin (Inactive) made changes -
            Summary Assertion failure in thread 139772003768576 in file srv0start.cc line 1475 backup fails with custom innodb_undo_pablespaces
            anikitin Andrii Nikitin (Inactive) made changes -
            Summary backup fails with custom innodb_undo_pablespaces backup fails with custom innodb_undo_pablespaces prev_space_id + 1 == undo_tablespace_ids[i]
            wlad Vladislav Vaintroub made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            wlad Vladislav Vaintroub made changes -
            anikitin Andrii Nikitin (Inactive) made changes -
            Summary backup fails with custom innodb_undo_pablespaces prev_space_id + 1 == undo_tablespace_ids[i] backup fails with custom innodb_undo_tablespaces prev_space_id + 1 == undo_tablespace_ids[i]
            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            10.2.7 shows little different assert

            2017-07-13 15:10:40 0x7f3d70464740  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc line 960
            InnoDB: Failing assertion: undo_tablespace_ids[i] != 0
            InnoDB: We intentionally generate a memory trap.
            

            anikitin Andrii Nikitin (Inactive) added a comment - - edited 10.2.7 shows little different assert 2017-07-13 15:10:40 0x7f3d70464740 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/srv/srv0start.cc line 960 InnoDB: Failing assertion: undo_tablespace_ids[i] != 0 InnoDB: We intentionally generate a memory trap.
            wlad Vladislav Vaintroub made changes -
            Assignee Vladislav Vaintroub [ wlad ] Marko Mäkelä [ marko ]
            wlad Vladislav Vaintroub made changes -
            mxu Michael Xu added a comment -

            my.cnf attached, and I think the error may caused by undo_tablespaces setting

            # this is only for the mysqld standalone daemon
            [mysqld]
            innodb_undo_directory                 = /var/lib/mysql                 # 
            innodb_undo_tablespaces               = 12                             # 
            innodb_undo_logs                      = 128                            # 
            innodb_max_undo_log_size              = 1G                             # 
            innodb_undo_log_truncate              = 1                              #
            [mariadb]
            [mariadb-10.1]
            

            mxu Michael Xu added a comment - my.cnf attached, and I think the error may caused by undo_tablespaces setting # this is only for the mysqld standalone daemon [mysqld] innodb_undo_directory = /var/lib/mysql # innodb_undo_tablespaces = 12 # innodb_undo_logs = 128 # innodb_max_undo_log_size = 1G # innodb_undo_log_truncate = 1 # [mariadb] [mariadb- 10.1 ]

            MariaDB 10.1.24 was the first one to contain the merge from MySQL 5.6.36, which introduced
            Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES
            which is likely causing this issue. We must adapt Mariabackup to this change.

            marko Marko Mäkelä added a comment - MariaDB 10.1.24 was the first one to contain the merge from MySQL 5.6.36 , which introduced Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES which is likely causing this issue. We must adapt Mariabackup to this change.
            anikitin Andrii Nikitin (Inactive) made changes -
            Fix Version/s 10.2 [ 14601 ]

            I can trivially repeat with

            ./mtr --mysqld=--innodb-undo-tablespaces=4 --suite=mariabackup
            

            on both 10.1 and 10.2.

            marko Marko Mäkelä added a comment - I can trivially repeat with ./mtr --mysqld=--innodb-undo-tablespaces=4 --suite=mariabackup on both 10.1 and 10.2.

            The problem is that in backup mode, srv_undo_tablespaces_init() will not initialize the undo_tablespace_ids[]. In normal startup, trx_rseg_get_n_undo_tablespaces() would take care of it. During backup, we do not have a buffer pool, and we cannot access the TRX_SYS page directly in the system tablespace, because the file page could be stale (some changes for it could be in the redo log only). Because backup does not really parse individual redo log records, and because we need to discover all tablespaces (including the undo tablespaces) at the start of the backup, some other mechanism is needed.

            As of today, Percona Xtrabackup 2.3 is still based on MySQL 5.6.14, so it is missing any adjustment for the MySQL 5.6.36 change
            Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES
            But, it does initialize the undo_tablespace_ids[] to a sequence 1,2,3… (should be srv_undo_space_id_start,srv_undo_space_id_start+1,… with the above change).

            Mariabackup should read the first page of the undo001 file and determine the tablespace ID from there. Perhaps it could simply treat all the undo* files in the same way as *.ibd files.

            marko Marko Mäkelä added a comment - The problem is that in backup mode, srv_undo_tablespaces_init() will not initialize the undo_tablespace_ids[]. In normal startup, trx_rseg_get_n_undo_tablespaces() would take care of it. During backup, we do not have a buffer pool, and we cannot access the TRX_SYS page directly in the system tablespace, because the file page could be stale (some changes for it could be in the redo log only). Because backup does not really parse individual redo log records, and because we need to discover all tablespaces (including the undo tablespaces) at the start of the backup, some other mechanism is needed. As of today, Percona Xtrabackup 2.3 is still based on MySQL 5.6.14, so it is missing any adjustment for the MySQL 5.6.36 change Bug #25551311 BACKPORT BUG #23517560 REMOVE SPACE_ID RESTRICTION FOR UNDO TABLESPACES But, it does initialize the undo_tablespace_ids[] to a sequence 1,2,3… (should be srv_undo_space_id_start,srv_undo_space_id_start+1,… with the above change). Mariabackup should read the first page of the undo001 file and determine the tablespace ID from there. Perhaps it could simply treat all the undo* files in the same way as *.ibd files.
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -

            To reduce the scope of this fix, I filed a separate bug for the 5.6.36 incompatibility:
            MDEV-13561 Mariabackup is incompatible with retroactively created innodb_undo_tablespaces

            marko Marko Mäkelä added a comment - To reduce the scope of this fix, I filed a separate bug for the 5.6.36 incompatibility: MDEV-13561 Mariabackup is incompatible with retroactively created innodb_undo_tablespaces
            marko Marko Mäkelä made changes -
            Summary backup fails with custom innodb_undo_tablespaces prev_space_id + 1 == undo_tablespace_ids[i] backup fails if innodb_undo_tablespaces>0
            marko Marko Mäkelä made changes -
            Fix Version/s 10.1.27 [ 22609 ]
            Fix Version/s 10.2.8 [ 22544 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.1 [ 16100 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.2.9 [ 22611 ]
            Fix Version/s 10.2.8 [ 22544 ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.3.2 [ 22533 ]
            mg MG made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 81075 ] MariaDB v4 [ 152264 ]

            People

              marko Marko Mäkelä
              mxu Michael Xu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.