[MDEV-15780] Mariabackup fails on --prepare stage when innodb_data_file_path contains a path Created: 2018-04-04  Updated: 2020-08-25  Resolved: 2018-04-12

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.1, 10.2, 10.3
Fix Version/s: 10.1.33, 10.2.15

Type: Bug Priority: Major
Reporter: Chris Calender (Inactive) Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None


 Description   

If you set innodb_data_file_path to include a path, then the --prepare step fails when using mariabackup.

For instance, suppose you have the following 2 items set:

datadir= /var/lib/mysql
 
innodb_data_file_path=/var/lib/mysql/ibdata1:12M:autoextend

Now, if you run mariabackup with --backup, it will run fine.

However, when you attempt mariabackup with --prepare, it fails with:

chris@chris-linux-laptop-64:~/Desktop$ sudo mariabackup --prepare --target-dir=/mysqlbackup 
mariabackup based on MariaDB server 10.2.14-MariaDB debian-linux-gnu (x86_64) 
mariabackup: cd to /mysqlbackup/
 
mariabackup: This target seems to be not prepared yet.
 
mariabackup: using the following InnoDB configuration for recovery:
 
mariabackup:   innodb_data_home_dir = .
 
mariabackup:   innodb_data_file_path = /var/lib/mysql/ibdata1:12M:autoextend
 
mariabackup:   innodb_log_group_home_dir = .
 
mariabackup: Starting InnoDB instance for recovery.
 
mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Uses event mutexes
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Number of pools: 1
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Using SSE2 crc32 instructions
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: Completed initialization of buffer pool
 
2018-04-04 14:38:39 139954027566848 [Note] InnoDB: page_cleaner coordinator priority: -20
 
2018-04-04 14:38:39 139954270910336 [Note] InnoDB: The first innodb_system data file '/var/lib/mysql/ibdata1' did not exist. A new tablespace will be created!
 
2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: Operating system error number 2 in a file operation.
 
2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: The error means the system cannot find the path specified.
 
2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
 
2018-04-04 14:38:39 139954270910336 [ERROR] InnoDB: File .//var/lib/mysql/ibdata1: 'create' returned OS error 71. Cannot continue operation
 
180404 14:38:39 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.14-MariaDB-10.2.14+maria~trusty
 
key_buffer_size=0
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=1
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
addr2line: 'mariabackup': No such file
 
mariabackup(my_print_stacktrace+0x2e)[0x7f49a55e4a4e]
 
mariabackup(handle_fatal_signal+0x355)[0x7f49a50d3175]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f49a4599340]
 
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f49a2950f79]
 
linux/raise.c:56(__GI_raise)[0x7f49a2954388]
 
addr2line: 'mariabackup': No such file
 
mariabackup(+0x9abe25)[0x7f49a5377e25]
 
mariabackup(+0x8f8735)[0x7f49a52c4735]
 
mariabackup(+0x8fdf9a)[0x7f49a52c9f9a]
 
mariabackup(+0xa4a97c)[0x7f49a541697c]
 
mariabackup(+0x458252)[0x7f49a4e24252]
 
mariabackup(main+0x185)[0x7f49a4e02b95]
 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f49a293bec5]
 
addr2line: 'mariabackup': No such file
 
mariabackup(+0x4501fd)[0x7f49a4e1c1fd]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.

If you just have innodb_data_file_path set to:

innodb_data_file_path=ibdata1:12M:autoextend

Then it works as expected.

Alternatively, if you edit the backup-my.cnf file that is generated in the --target-dir and remove the path from innodb_data_file_path, then it also works as expected.

This extra step is not needed in Xtrabackup.

 Comments   
Comment by Vladislav Vaintroub [ 2018-04-11 ]

ccalender, with absolute paths in innodb_data_file_path, I cannot get mysqld to start. this is what it tells me,on Linux

2018-04-11  9:05:25 0 [ERROR] InnoDB: File .//home/wlad/ibdata1: 'create' returned OS error 71. Cannot continue operatio
 
n
 
180411  9:05:25 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.

Windows is similar. To get a backup , one would first need a server started with such parameter, and that does not quite work. I'm not sure how it is supposed to work. Any idea?

Comment by Chris Calender (Inactive) [ 2018-04-11 ]

Hi wlad,

There is a way to get this to work on Linux. I initially tried with Windows, since that is what I had most handy, and encountered the same issue as you above. However, when I switched to Linux (Ubuntu, specifically), I was able to get mysqld started and then run mariabackup (as you can see from the results). This is also what the other reporter did (and why they informed me about the bug).

To keep things simple, I first started mysqld with the following:

datadir= /var/lib/mysql
 
innodb_data_file_path=ibdata1:12M:autoextend

Once that worked, I then modified innodb_data_file_path to:

datadir= /var/lib/mysql
 
innodb_data_file_path=/var/lib/mysql/ibdata1:12M:autoextend

I did not have any other path settings in effect, such as innodb_data_home_dir and innodb_log_group_home_dir.

I did some more testing last night, and will be filing a separate bug today about how mysqld will not even start on Windows if you give it a full path starting with "C:..". Perhaps the two are involved, but again, both the reporter and I were able to get mysqld started with these values on Linux, and thus able to produce the mariabackup bug I reported here.

Comment by Vladislav Vaintroub [ 2018-04-11 ]

my attempt was from Linux (.//home/wlad/ibdata1 shows it), ubuntu specifically, after I failed to get it working from Windows prior to that.

I set datadir = to my $HOME (which is /home/wlad),
What do I do wrong?
marko says btw, full paths are not supposed to work. With innodb, and by extension, with backup

Comment by Vladislav Vaintroub [ 2018-04-11 ]

Does not quite work.

if I do

sql/mysqld --no-defaults --lc-messages-dir=/home/mariadb/wlad/1 0.2/xxx/sql/share --datadir=$HOME --innodb-data-file-path=$HOME/ibdata1:10M:auto extend --skip-grant-tables

I get 

2018-04-11 18:10:51 140303482238784 [Note] sql/mysqld (mysqld 10.2.15-MariaDB-debug) starting as process 11128 ...
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Uses event mutexes
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Compressed tables use zlib 1.2.8
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using Linux native AIO
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Number of pools: 1
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Using SSE2 crc32 instructions
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: Completed initialization of buffer pool
 
2018-04-11 18:10:51 140302758807296 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 
2018-04-11 18:10:51 140303482238784 [Note] InnoDB: The first innodb_system data file '/home/mariadb/ibdata1' did not exist. A new tablespace will be created!
 
2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: Operating system error number 2 in a file operation.
 
2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: The error means the system cannot find the path specified.
 
2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
 
2018-04-11 18:10:51 140303482238784 [ERROR] InnoDB: File .//home/mariadb/ibdata1: 'create' returned OS error 71. Cannot continue operation
 
180411 18:10:51 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.15-MariaDB-debug
 
key_buffer_size=134217728
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=153
 
thread_count=0
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467349 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...

Comment by Vladislav Vaintroub [ 2018-04-11 ]

Ok, got it empty innodb-data-home-dir is necessary as well.

Generated at Thu Feb 08 08:23:58 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.