[MDEV-15779] mariabackup --apply-log-only fails if underlying file system is a CIFS mount Created: 2018-04-04  Updated: 2024-01-23  Resolved: 2018-04-12

Status: Closed
Project: MariaDB Server
Component/s: Backup
Affects Version/s: 10.2.13
Fix Version/s: 10.1.33, 10.2.15

Type: Bug Priority: Major
Reporter: Rick Pizzi Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None
Environment:

Ubuntu

Attachments: File mariabackup_strace.txt.gz    

 Description   

When trying to restore a backup that was taken with same version of mariabackup, and that includes incremental pieces, the restore fails when we try to apply the first incremental
to the base (full) backup.

We found out that this is tied to the fact the underlying filesystem is a CIFS mount.
The exact same operation with same data succeeds if we try this on a local filesystem.

CIFS mount options: 

/Production-Backups/prod-mariadb03 on /backup type cifs (rw,relatime,vers=2.0,cache=strict,username=svc-backup-prod,domain=redacted,uid=0,noforceuid,gid=0,noforcegid,addr=10.15.200.31,file_mode=0775,dir_mode=0775,nounix,serverino,mapposix,rsize=65536,wsize=65536,echo_interval=60,actimeo=1)

Command line used to apply incremental:

$XTRABACKUP --defaults-file=$TD/backup-my.cnf --prepare --use-memory=4G --apply-log-only --target-dir=$TD --incremental-dir=$TD/incrementals/$piece

Failure log:

mariabackup based on MariaDB server 10.2.13-MariaDB debian-linux-gnu (x86_64) 
incremental backup from 1875122129 is enabled.
 
mariabackup: cd to /backup/bgbackup-maria/rick/
 
mariabackup: This target seems to be already prepared.
 
mariabackup: using the following InnoDB configuration for recovery:
 
mariabackup:   innodb_data_home_dir = .
 
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
 
mariabackup:   innodb_log_group_home_dir = /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/
 
mariabackup: Generating a list of tablespaces
 
mariabackup: page size for /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta is 16384 bytes
 
Applying /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta to ./ibdata1...
 
2018-04-04 17:32:59 140111916681408 [Warning] InnoDB: Retry attempts for reading partial data failed.
 
2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Tried to read 16384 bytes at offset 0, but was only able to read 0
 
2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Operating system error number 22 in a file operation.
 
2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
 
2018-04-04 17:32:59 140111916681408 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
 
2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: File (unknown): 'read' returned OS error 222. Cannot continue operation
 
180404 17:32:59 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.2.13-MariaDB-10.2.13+maria~xenial
 
key_buffer_size=0
 
read_buffer_size=131072
 
max_used_connections=0
 
max_threads=1
 
thread_count=0
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x0 thread_stack 0x49000
 
addr2line: 'mariabackup': No such file
 
mariabackup(my_print_stacktrace+0x2e)[0x55e29c08481e]
 
mariabackup(handle_fatal_signal+0x345)[0x55e29bb81835]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6e58cc4390]
 
linux/raise.c:54(__GI_raise)[0x7f6e56f6b428]
 
stdlib/abort.c:91(__GI_abort)[0x7f6e56f6d02a]
 
addr2line: 'mariabackup': No such file
 
mariabackup(+0x9fd6b6)[0x55e29be906b6]
 
mariabackup(+0xa030f8)[0x55e29be960f8]
 
mariabackup(+0xa031b2)[0x55e29be961b2]
 
mariabackup(+0x4324dc)[0x55e29b8c54dc]
 
mariabackup(+0x408cb7)[0x55e29b89bcb7]
 
mariabackup(+0x437053)[0x55e29b8ca053]
 
mariabackup(main+0x185)[0x55e29b8ad145]
 
csu/libc-start.c:325(__libc_start_main)[0x7f6e56f56830]
 
addr2line: 'mariabackup': No such file
 
mariabackup(_start+0x29)[0x55e29b8c1f79]
 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
information that should help you find out what is causing the crash.



 Comments   
Comment by Vladislav Vaintroub [ 2018-04-04 ]

can you check the size of ibdata1.delta?
also could you please provide strace of your attempt.

Comment by Rick Pizzi [ 2018-04-04 ] 

ls -l /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/
 
total 8091
 
-rwxrwxr-x 1 root root   16384 Apr  4 17:26 aria_log.00000001
 
-rwxrwxr-x 1 root root      52 Apr  4 17:26 aria_log_control
 
-rwxrwxr-x 1 root root     298 Apr  4 17:26 backup-my.cnf
 
-rwxrwxr-x 1 root root   62359 Apr  4 17:26 ib_buffer_pool
 
-rwxrwxr-x 1 root root 8192000 Apr  4 17:26 ibdata1.delta
 
-rwxrwxr-x 1 root root      45 Apr  4 01:29 ibdata1.meta
 
-rwxrwxr-x 1 root root    2560 Apr  4 17:26 ib_logfile0
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 mariadb
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 maxscale_schema
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 mdbutil
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 mysql
 
drwxrwxr-x 2 root root       0 Apr  4 01:29 performance_schema
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_customers
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_entities
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_identityserver
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_idmapper
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_logging
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_markettemplates
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_oddsprocessing
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_punters
 
drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_risk
 
-rwxrwxr-x 1 root root      39 Apr  4 17:26 xtrabackup_binlog_info
 
-rwxrwxr-x 1 root root     114 Apr  4 01:29 xtrabackup_checkpoints
 
-rwxrwxr-x 1 root root     770 Apr  4 17:26 xtrabackup_info
 
-rwxrwxr-x 1 root root      87 Apr  4 17:26 xtrabackup_slave_info

Comment by Rick Pizzi [ 2018-04-04 ]

strace output attached!

Comment by Vladislav Vaintroub [ 2018-04-04 ]

This is the sequence that generates an error and it should not
File opened OK, some fcntl set OK, fadvise also OK
but pread fails, with valid file descriptor, valid size, valid memory.

open("/backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta", O_RDWR) = 4
 
fcntl(4, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
 
fadvise64(4, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
 
fcntl(4, F_SETFL, O_RDONLY|O_DIRECT)    = 0
 
mmap(NULL, 67129344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9273b75000
 
pread64(4, 0x7f9273b78000, 16384, 0)    = -1 EINVAL (Invalid argument)

Comment by Vladislav Vaintroub [ 2018-04-04 ]

I suggest to try forcedirectio option for mounting, to workaround CIFS bug
https://www.cyberciti.biz/tips/disable-caching-on-the-cifs-nfs-client.html

Comment by Rick Pizzi [ 2018-04-04 ]

Can't change mount option for the share, it's .... shared with other applications....
But I understand this fails outside mariabackup.
Could be useful to mention this incompatibility in the documentation maybe?

Comment by Claudio Nanni [ 2024-01-23 ]

One case has been solved by mounting the CIFS with cache=none option.

Generated at Thu Feb 08 08:23:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.