Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13499

Backing up table that "doesn't exist in engine" cause crash in mariabackup when using encryption

Details

    • 10.1.31

    Description

      When there is file for a table in a database directory in the datadir which is not in the data dictionary for InnoDB and tables are encrypted, then running mariaback will crash mariabackup with a core dump.
      How to reproduce:
      Install MariaDB 10.1.26 and MariaDB-Backup
      $ sudo bash

      1. service mysql start
      2. mysql -u root test -e "CREATE TABLE t1(c1 INT)"
      3. mysql -u root test -e "INSERT INTO t1 VALUES(1)"
      4. service mysql stop
      5. cd /var/lib/mysql/test
      6. cp t1.frm t1_1.frm
      7. cp t1.ibd t1_1.ibd
      8. chown mysql:mysql t1_1.*
      9. service mysql start
      10. mkdir /tmp/backup
      11. mariabackup --backup --target-dir=/tmp/backup/ --user=root
        Crash!

      Attachments

        Issue Links

          Activity

            The output from mariabackup is as follow when the crash happens:
            170817 15:18:42 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
            Using server version 10.1.26-MariaDB
            Loading encryption plugin
            Encryption plugin parameter : '--file_key_management_encryption_algorithm=aes_cbc'
            Encryption plugin parameter : '--file_key_management_filekey='
            Encryption plugin parameter : '--file_key_management_filename=/var/lib/mysql/keys'
            mariabackup based on MariaDB server 10.1.26-MariaDB Linux (x86_64)
            xtrabackup: uses posix_fadvise().
            xtrabackup: cd to /var/lib/mysql/
            xtrabackup: open files limit requested 0, set to 1024
            xtrabackup: using the following InnoDB configuration:
            xtrabackup: innodb_data_home_dir = ./
            xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
            xtrabackup: innodb_log_group_home_dir = ./
            xtrabackup: innodb_log_files_in_group = 2
            xtrabackup: innodb_log_file_size = 50331648
            170817 15:18:42 >> log scanned up to (51414205)
            xtrabackup: Generating a list of tablespaces
            InnoDB: Attempted to open a previously opened tablespace. Previous tablespace test/t1_1 uses space ID: 8 at filepath: ./test/t1_1.ibd. Cannot open tablespace test/t1 which uses space ID: 8 at filepath: ./test/t1.ibd
            2017-08-17 15:18:42 7f86606da820 InnoDB: Operating system error number 2 in a file operation.
            InnoDB: The error means the system cannot find the path specified.
            InnoDB: If you are installing InnoDB, remember that you must create
            InnoDB: directories yourself, InnoDB does not create them.
            InnoDB: Error: could not open single-table tablespace file ./test/t1.ibd
            InnoDB: We do not continue the crash recovery, because the table may become
            InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
            InnoDB: To fix the problem and start mysqld:
            InnoDB: 1) If there is a permission problem in the file and mysqld cannot
            InnoDB: open the file, you should modify the permissions.
            InnoDB: 2) If the table is not needed, or you can restore it from a backup,
            InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
            InnoDB: crash recovery and ignore that table.
            InnoDB: 3) If the file system or the disk is broken, and you cannot remove
            InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
            InnoDB: and force InnoDB to continue crash recovery here.
            170817 15:18:42 [ERROR] mysqld got signal 6 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.1.26-MariaDB
            key_buffer_size=0
            read_buffer_size=131072
            max_used_connections=0
            max_threads=1
            thread_count=0
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5309 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x0 thread_stack 0x48400
            addr2line: 'mariabackup': No such file
            mariabackup(my_print_stacktrace+0x2b)[0x7f866119fa5b]
            mariabackup(handle_fatal_signal+0x4d5)[0x7f8660d02895]
            /lib64/libpthread.so.0(+0x31d180f710)[0x7f86602c9710]
            /lib64/libc.so.6(gsignal+0x35)[0x7f865e6f6925]
            /lib64/libc.so.6(abort+0x175)[0x7f865e6f8105]
            mariabackup(+0x881a79)[0x7f8660f7aa79]
            mariabackup(+0x3d6951)[0x7f8660acf951]
            mariabackup(_Z22xtrabackup_backup_funcv+0xc9a)[0x7f8660ad52ea]
            mariabackup(main+0xf77)[0x7f8660ab4f77]
            /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f865e6e2d1d]
            mariabackup(+0x3d3b29)[0x7f8660accb29]
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.

            karlsson Anders Karlsson added a comment - The output from mariabackup is as follow when the crash happens: 170817 15:18:42 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set Using server version 10.1.26-MariaDB Loading encryption plugin Encryption plugin parameter : '--file_key_management_encryption_algorithm=aes_cbc' Encryption plugin parameter : '--file_key_management_filekey=' Encryption plugin parameter : '--file_key_management_filename=/var/lib/mysql/keys' mariabackup based on MariaDB server 10.1.26-MariaDB Linux (x86_64) xtrabackup: uses posix_fadvise(). xtrabackup: cd to /var/lib/mysql/ xtrabackup: open files limit requested 0, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb_data_home_dir = ./ xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = ./ xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 170817 15:18:42 >> log scanned up to (51414205) xtrabackup: Generating a list of tablespaces InnoDB: Attempted to open a previously opened tablespace. Previous tablespace test/t1_1 uses space ID: 8 at filepath: ./test/t1_1.ibd. Cannot open tablespace test/t1 which uses space ID: 8 at filepath: ./test/t1.ibd 2017-08-17 15:18:42 7f86606da820 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. InnoDB: Error: could not open single-table tablespace file ./test/t1.ibd InnoDB: We do not continue the crash recovery, because the table may become InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it. InnoDB: To fix the problem and start mysqld: InnoDB: 1) If there is a permission problem in the file and mysqld cannot InnoDB: open the file, you should modify the permissions. InnoDB: 2) If the table is not needed, or you can restore it from a backup, InnoDB: then you can remove the .ibd file, and InnoDB will do a normal InnoDB: crash recovery and ignore that table. InnoDB: 3) If the file system or the disk is broken, and you cannot remove InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf InnoDB: and force InnoDB to continue crash recovery here. 170817 15:18:42 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.26-MariaDB key_buffer_size=0 read_buffer_size=131072 max_used_connections=0 max_threads=1 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5309 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 addr2line: 'mariabackup': No such file mariabackup(my_print_stacktrace+0x2b) [0x7f866119fa5b] mariabackup(handle_fatal_signal+0x4d5) [0x7f8660d02895] /lib64/libpthread.so.0(+0x31d180f710) [0x7f86602c9710] /lib64/libc.so.6(gsignal+0x35) [0x7f865e6f6925] /lib64/libc.so.6(abort+0x175) [0x7f865e6f8105] mariabackup(+0x881a79) [0x7f8660f7aa79] mariabackup(+0x3d6951) [0x7f8660acf951] mariabackup(_Z22xtrabackup_backup_funcv+0xc9a) [0x7f8660ad52ea] mariabackup(main+0xf77) [0x7f8660ab4f77] /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f865e6e2d1d] mariabackup(+0x3d3b29) [0x7f8660accb29] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.

            "Crash with coredump" is an assertion. It tells at least what was wrong, so DBA can fix the manually manipulated files, and do his backup again

            wlad Vladislav Vaintroub added a comment - "Crash with coredump" is an assertion. It tells at least what was wrong, so DBA can fix the manually manipulated files, and do his backup again

            in my.cnf the following myseld options were set:
            plugin-load-add=file_key_management
            file_key_management_encryption_algorithm=aes_cbc
            file_key_management_filename = /var/lib/mysql/keys
            innodb-encrypt-log=ON
            innodb-encrypt-tables=FORCE
            encrypt-binlog=ON

            karlsson Anders Karlsson added a comment - in my.cnf the following myseld options were set: plugin-load-add=file_key_management file_key_management_encryption_algorithm=aes_cbc file_key_management_filename = /var/lib/mysql/keys innodb-encrypt-log=ON innodb-encrypt-tables=FORCE encrypt-binlog=ON
            anikitin Andrii Nikitin (Inactive) added a comment - - edited

            Thank you for report - I consider it helped in verification of MDEV-13284 and according to my observations the problem exists in upstream InnoDB as well. (maybe encryption helps to trigger the issue, but it was repeated without encryption as well)
            Let's consider this ticket as duplicate of MDEV-13284 until latter is fixed. Then we just check if the problem still reproduces in mariabackup after fix of MDEV-13284 .

            anikitin Andrii Nikitin (Inactive) added a comment - - edited Thank you for report - I consider it helped in verification of MDEV-13284 and according to my observations the problem exists in upstream InnoDB as well. (maybe encryption helps to trigger the issue, but it was repeated without encryption as well) Let's consider this ticket as duplicate of MDEV-13284 until latter is fixed. Then we just check if the problem still reproduces in mariabackup after fix of MDEV-13284 .

            commit 2749d2509661c2addbf78ec5f5b2218ff7301b0f
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Mon Jan 29 14:21:08 2018 +0200

            MDEV-13499: Backing up table that "doesn't exist in engine" cause crash in mariabackup when using encryption

            Problem was that there is intentional crah when .ibd file does
            not found. In mariabackup case we should avoid this crash.

            jplindst Jan Lindström (Inactive) added a comment - commit 2749d2509661c2addbf78ec5f5b2218ff7301b0f Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Mon Jan 29 14:21:08 2018 +0200 MDEV-13499 : Backing up table that "doesn't exist in engine" cause crash in mariabackup when using encryption Problem was that there is intentional crah when .ibd file does not found. In mariabackup case we should avoid this crash.

            People

              jplindst Jan Lindström (Inactive)
              karlsson Anders Karlsson
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.