[MDEV-30266] Preparing a mariabackup on a mounted CIFS / SMB file system fails Created: 2022-12-19  Updated: 2023-11-28

Status: Open
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.6.11, 10.7.7, 10.8.6, 10.9.4, 10.10.2, 10.11.1
Fix Version/s: 10.6, 10.11

Type: Bug Priority: Major
Reporter: Hartmut Holzgraefe Assignee: Alexander Barkov
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-26970 MariaDB InnoDB Plugin Failure - NUC/C... Confirmed
relates to MDEV-26995 mariabackup --prepare fails on Window... Closed
relates to MDEV-26970 MariaDB InnoDB Plugin Failure - NUC/C... Confirmed

 Description   

Running the --backup stage of mariabackup with the target-dir being on a mounted CIFS share works fine, but the --prepare stage then fails early with an IO error:

root@ubuntu-jammy:~# strace -f -o log -s10000  mariabackup --prepare --target-dir=/mnt/X
 
mariabackup based on MariaDB server 10.9.4-MariaDB debian-linux-gnu (x86_64)
 
[00] 2022-12-19 07:59:12 cd to /mnt/X/
 
[00] 2022-12-19 07:59:12 open files limit requested 0, set to 1024
 
[00] 2022-12-19 07:59:12 This target seems to be not prepared yet.
 
[00] 2022-12-19 07:59:12 mariabackup: using the following InnoDB configuration for recovery:
 
[00] 2022-12-19 07:59:12 innodb_data_home_dir = .
 
[00] 2022-12-19 07:59:12 innodb_data_file_path = ibdata1:12M:autoextend
 
[00] 2022-12-19 07:59:12 innodb_log_group_home_dir = .
 
[00] 2022-12-19 07:59:12 Starting InnoDB instance for recovery.
 
[00] 2022-12-19 07:59:12 mariabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
 
2022-12-19  7:59:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2022-12-19  7:59:12 0 [Note] InnoDB: Number of transaction pools: 1
 
2022-12-19  7:59:12 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2022-12-19  7:59:12 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000MiB, chunk size = 100.000MiB
 
2022-12-19  7:59:12 0 [Note] InnoDB: Completed initialization of buffer pool
 
[00] 2022-12-19 07:59:12 mariadb-backup: srv_start() returned 100 (I/O error).

strace shows that the ibdata1 file can be opened just fine, but reading from it fails with "invalid argument"

2434  openat(AT_FDCWD, "./ibdata1", O_RDWR|O_CLOEXEC <unfinished ...>
 
2434  <... openat resumed>)             = 6
 
2434  fcntl(6, F_SETFL, O_RDONLY|O_DIRECT <unfinished ...>
 
2434  <... fcntl resumed>)              = 0
 
2434  fcntl(6, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
 
2434  newfstatat(6, "", {st_mode=S_IFREG|0755, st_size=12582912, ...}, AT_EMPTY_PATH) = 0
 
2434  pread64(6, 0x55af89b64000, 65536, 0) = -1 EINVAL (Invalid argument)
 
2434  newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=114, ...}, 0) = 0
 
2434  write(2, "[00] 2022-12-19 07:59:12 mariadb-backup: srv_start() returned 100 (I/O error).\n", 79) = 79

For testing I used a Samba 4.16 server on Linux as the CIFS server.

 Comments   
Comment by Hartmut Holzgraefe [ 2022-12-19 ]

Turns out this only started with 10.6, earlier versions can prepare the backup on the CIFS share just fine

Comment by Hartmut Holzgraefe [ 2022-12-19 ]

Why "Fix for" for 10.3 to 10.5? This only started with 10.6, earlier versions work fine. So it should only be "Fix for" for 10.6 and above?

Comment by Daniel Black [ 2022-12-19 ]

I think this is in the general class of bugs that O_DIRECT and CIFS is rather specific about the mount option and the kernel versions.

From MDEV-26970, cache=none on the mount options might help or innodb_flush_method=fsync.

Comment by Hartmut Holzgraefe [ 2022-12-19 ]

cache=none indeed seems to do the trick

not checked innodb_flush_method yet, but as that changed its default value in 10.6 that would explain why I was only hitting it in 10.6+

Comment by Hartmut Holzgraefe [ 2022-12-20 ]

--innnodb-flush-method=fsync on the command line also works fine.

I couldn't get it to read such setting from my.cnf though.

When trying 

[mariabackup]
 
innnodb-flush-method=fsync

I'm just getting the same error, and the variables entry in the --help output still shows O_DIRECT

When trying 

[server]
 
innnodb-flush-method=fsync

I see the variable value in --help output change to "fsync", but when running --prepare it still runs into the same error.

That's probably material for a separate jira ticket though ...

Generated at Thu Feb 08 10:14:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.