[MDEV-13499] Backing up table that "doesn't exist in engine" cause crash in mariabackup when using encryption Created: 2017-08-11  Updated: 2018-01-29  Resolved: 2018-01-29

Status: Closed
Project: MariaDB Server
Component/s: Backup, Storage Engine - InnoDB
Affects Version/s: 10.1
Fix Version/s: 10.1.31

Type: Bug Priority: Major
Reporter: Anders Karlsson Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

Linux CentOS 6.5


Issue Links:
Duplicate
is duplicated by MDEV-13284 mariadb-server: fil_load_single_table... Closed
Sprint: 10.1.31

 Description   

When there is file for a table in a database directory in the datadir which is not in the data dictionary for InnoDB and tables are encrypted, then running mariaback will crash mariabackup with a core dump.
How to reproduce:
Install MariaDB 10.1.26 and MariaDB-Backup
$ sudo bash

  1. service mysql start
  2. mysql -u root test -e "CREATE TABLE t1(c1 INT)"
  3. mysql -u root test -e "INSERT INTO t1 VALUES(1)"
  4. service mysql stop
  5. cd /var/lib/mysql/test
  6. cp t1.frm t1_1.frm
  7. cp t1.ibd t1_1.ibd
  8. chown mysql:mysql t1_1.*
  9. service mysql start
  10. mkdir /tmp/backup
  11. mariabackup --backup --target-dir=/tmp/backup/ --user=root
    Crash!


 Comments   
Comment by Anders Karlsson [ 2017-08-17 ]

The output from mariabackup is as follow when the crash happens:
170817 15:18:42 Connecting to MySQL server host: localhost, user: root, password: not set, port: not set, socket: not set
Using server version 10.1.26-MariaDB
Loading encryption plugin
Encryption plugin parameter : '--file_key_management_encryption_algorithm=aes_cbc'
Encryption plugin parameter : '--file_key_management_filekey='
Encryption plugin parameter : '--file_key_management_filename=/var/lib/mysql/keys'
mariabackup based on MariaDB server 10.1.26-MariaDB Linux (x86_64)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql/
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
170817 15:18:42 >> log scanned up to (51414205)
xtrabackup: Generating a list of tablespaces
InnoDB: Attempted to open a previously opened tablespace. Previous tablespace test/t1_1 uses space ID: 8 at filepath: ./test/t1_1.ibd. Cannot open tablespace test/t1 which uses space ID: 8 at filepath: ./test/t1.ibd
2017-08-17 15:18:42 7f86606da820 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: Error: could not open single-table tablespace file ./test/t1.ibd
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
170817 15:18:42 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.26-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5309 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
addr2line: 'mariabackup': No such file
mariabackup(my_print_stacktrace+0x2b)[0x7f866119fa5b]
mariabackup(handle_fatal_signal+0x4d5)[0x7f8660d02895]
/lib64/libpthread.so.0(+0x31d180f710)[0x7f86602c9710]
/lib64/libc.so.6(gsignal+0x35)[0x7f865e6f6925]
/lib64/libc.so.6(abort+0x175)[0x7f865e6f8105]
mariabackup(+0x881a79)[0x7f8660f7aa79]
mariabackup(+0x3d6951)[0x7f8660acf951]
mariabackup(_Z22xtrabackup_backup_funcv+0xc9a)[0x7f8660ad52ea]
mariabackup(main+0xf77)[0x7f8660ab4f77]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f865e6e2d1d]
mariabackup(+0x3d3b29)[0x7f8660accb29]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Comment by Vladislav Vaintroub [ 2017-08-17 ]

"Crash with coredump" is an assertion. It tells at least what was wrong, so DBA can fix the manually manipulated files, and do his backup again

Comment by Anders Karlsson [ 2017-08-17 ]

in my.cnf the following myseld options were set:
plugin-load-add=file_key_management
file_key_management_encryption_algorithm=aes_cbc
file_key_management_filename = /var/lib/mysql/keys
innodb-encrypt-log=ON
innodb-encrypt-tables=FORCE
encrypt-binlog=ON

Comment by Andrii Nikitin (Inactive) [ 2017-08-23 ]

Thank you for report - I consider it helped in verification of MDEV-13284 and according to my observations the problem exists in upstream InnoDB as well. (maybe encryption helps to trigger the issue, but it was repeated without encryption as well)
Let's consider this ticket as duplicate of MDEV-13284 until latter is fixed. Then we just check if the problem still reproduces in mariabackup after fix of MDEV-13284 .

Comment by Jan Lindström (Inactive) [ 2018-01-29 ]

commit 2749d2509661c2addbf78ec5f5b2218ff7301b0f
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Mon Jan 29 14:21:08 2018 +0200

MDEV-13499: Backing up table that "doesn't exist in engine" cause crash in mariabackup when using encryption

Problem was that there is intentional crah when .ibd file does
not found. In mariabackup case we should avoid this crash.

Generated at Thu Feb 08 08:06:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.