Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15779

mariabackup --apply-log-only fails if underlying file system is a CIFS mount

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.2.13
    • 10.1.33, 10.2.15
    • Backup
    • None
    • Ubuntu

    Description

      When trying to restore a backup that was taken with same version of mariabackup, and that includes incremental pieces, the restore fails when we try to apply the first incremental
      to the base (full) backup.

      We found out that this is tied to the fact the underlying filesystem is a CIFS mount.
      The exact same operation with same data succeeds if we try this on a local filesystem.

      CIFS mount options:

      /Production-Backups/prod-mariadb03 on /backup type cifs (rw,relatime,vers=2.0,cache=strict,username=svc-backup-prod,domain=redacted,uid=0,noforceuid,gid=0,noforcegid,addr=10.15.200.31,file_mode=0775,dir_mode=0775,nounix,serverino,mapposix,rsize=65536,wsize=65536,echo_interval=60,actimeo=1)
      

      Command line used to apply incremental:

      $XTRABACKUP --defaults-file=$TD/backup-my.cnf --prepare --use-memory=4G --apply-log-only --target-dir=$TD --incremental-dir=$TD/incrementals/$piece 
      

      Failure log:

      mariabackup based on MariaDB server 10.2.13-MariaDB debian-linux-gnu (x86_64) 
      incremental backup from 1875122129 is enabled.
      mariabackup: cd to /backup/bgbackup-maria/rick/
      mariabackup: This target seems to be already prepared.
      mariabackup: using the following InnoDB configuration for recovery:
      mariabackup:   innodb_data_home_dir = .
      mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      mariabackup:   innodb_log_group_home_dir = /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/
      mariabackup: Generating a list of tablespaces
      mariabackup: page size for /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta is 16384 bytes
      Applying /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta to ./ibdata1...
      2018-04-04 17:32:59 140111916681408 [Warning] InnoDB: Retry attempts for reading partial data failed.
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Tried to read 16384 bytes at offset 0, but was only able to read 0
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Operating system error number 22 in a file operation.
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
      2018-04-04 17:32:59 140111916681408 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: File (unknown): 'read' returned OS error 222. Cannot continue operation
      180404 17:32:59 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.13-MariaDB-10.2.13+maria~xenial
      key_buffer_size=0
      read_buffer_size=131072
      max_used_connections=0
      max_threads=1
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      addr2line: 'mariabackup': No such file
      mariabackup(my_print_stacktrace+0x2e)[0x55e29c08481e]
      mariabackup(handle_fatal_signal+0x345)[0x55e29bb81835]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6e58cc4390]
      linux/raise.c:54(__GI_raise)[0x7f6e56f6b428]
      stdlib/abort.c:91(__GI_abort)[0x7f6e56f6d02a]
      addr2line: 'mariabackup': No such file
      mariabackup(+0x9fd6b6)[0x55e29be906b6]
      mariabackup(+0xa030f8)[0x55e29be960f8]
      mariabackup(+0xa031b2)[0x55e29be961b2]
      mariabackup(+0x4324dc)[0x55e29b8c54dc]
      mariabackup(+0x408cb7)[0x55e29b89bcb7]
      mariabackup(+0x437053)[0x55e29b8ca053]
      mariabackup(main+0x185)[0x55e29b8ad145]
      csu/libc-start.c:325(__libc_start_main)[0x7f6e56f56830]
      addr2line: 'mariabackup': No such file
      mariabackup(_start+0x29)[0x55e29b8c1f79]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      

      Attachments

        Activity

          can you check the size of ibdata1.delta?
          also could you please provide strace of your attempt.

          wlad Vladislav Vaintroub added a comment - can you check the size of ibdata1.delta? also could you please provide strace of your attempt.

          ls -l /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/
          total 8091
          -rwxrwxr-x 1 root root   16384 Apr  4 17:26 aria_log.00000001
          -rwxrwxr-x 1 root root      52 Apr  4 17:26 aria_log_control
          -rwxrwxr-x 1 root root     298 Apr  4 17:26 backup-my.cnf
          -rwxrwxr-x 1 root root   62359 Apr  4 17:26 ib_buffer_pool
          -rwxrwxr-x 1 root root 8192000 Apr  4 17:26 ibdata1.delta
          -rwxrwxr-x 1 root root      45 Apr  4 01:29 ibdata1.meta
          -rwxrwxr-x 1 root root    2560 Apr  4 17:26 ib_logfile0
          drwxrwxr-x 2 root root       0 Apr  4 17:26 mariadb
          drwxrwxr-x 2 root root       0 Apr  4 17:26 maxscale_schema
          drwxrwxr-x 2 root root       0 Apr  4 17:26 mdbutil
          drwxrwxr-x 2 root root       0 Apr  4 17:26 mysql
          drwxrwxr-x 2 root root       0 Apr  4 01:29 performance_schema
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_customers
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_entities
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_identityserver
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_idmapper
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_logging
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_markettemplates
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_oddsprocessing
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_punters
          drwxrwxr-x 2 root root       0 Apr  4 17:26 prod_risk
          -rwxrwxr-x 1 root root      39 Apr  4 17:26 xtrabackup_binlog_info
          -rwxrwxr-x 1 root root     114 Apr  4 01:29 xtrabackup_checkpoints
          -rwxrwxr-x 1 root root     770 Apr  4 17:26 xtrabackup_info
          -rwxrwxr-x 1 root root      87 Apr  4 17:26 xtrabackup_slave_info
          

          rpizzi Rick Pizzi (Inactive) added a comment - ls -l /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/ total 8091 -rwxrwxr-x 1 root root 16384 Apr 4 17:26 aria_log.00000001 -rwxrwxr-x 1 root root 52 Apr 4 17:26 aria_log_control -rwxrwxr-x 1 root root 298 Apr 4 17:26 backup-my.cnf -rwxrwxr-x 1 root root 62359 Apr 4 17:26 ib_buffer_pool -rwxrwxr-x 1 root root 8192000 Apr 4 17:26 ibdata1.delta -rwxrwxr-x 1 root root 45 Apr 4 01:29 ibdata1.meta -rwxrwxr-x 1 root root 2560 Apr 4 17:26 ib_logfile0 drwxrwxr-x 2 root root 0 Apr 4 17:26 mariadb drwxrwxr-x 2 root root 0 Apr 4 17:26 maxscale_schema drwxrwxr-x 2 root root 0 Apr 4 17:26 mdbutil drwxrwxr-x 2 root root 0 Apr 4 17:26 mysql drwxrwxr-x 2 root root 0 Apr 4 01:29 performance_schema drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_customers drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_entities drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_identityserver drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_idmapper drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_logging drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_markettemplates drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_oddsprocessing drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_punters drwxrwxr-x 2 root root 0 Apr 4 17:26 prod_risk -rwxrwxr-x 1 root root 39 Apr 4 17:26 xtrabackup_binlog_info -rwxrwxr-x 1 root root 114 Apr 4 01:29 xtrabackup_checkpoints -rwxrwxr-x 1 root root 770 Apr 4 17:26 xtrabackup_info -rwxrwxr-x 1 root root 87 Apr 4 17:26 xtrabackup_slave_info

          strace output attached!

          rpizzi Rick Pizzi (Inactive) added a comment - strace output attached!

          This is the sequence that generates an error and it should not
          File opened OK, some fcntl set OK, fadvise also OK
          but pread fails, with valid file descriptor, valid size, valid memory.

          open("/backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta", O_RDWR) = 4
          fcntl(4, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
          fadvise64(4, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
          fcntl(4, F_SETFL, O_RDONLY|O_DIRECT)    = 0
          mmap(NULL, 67129344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9273b75000
          pread64(4, 0x7f9273b78000, 16384, 0)    = -1 EINVAL (Invalid argument)
          

          wlad Vladislav Vaintroub added a comment - This is the sequence that generates an error and it should not File opened OK, some fcntl set OK, fadvise also OK but pread fails, with valid file descriptor, valid size, valid memory. open("/backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta", O_RDWR) = 4 fcntl(4, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0 fadvise64(4, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 fcntl(4, F_SETFL, O_RDONLY|O_DIRECT) = 0 mmap(NULL, 67129344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9273b75000 pread64(4, 0x7f9273b78000, 16384, 0) = -1 EINVAL (Invalid argument)

          I suggest to try forcedirectio option for mounting, to workaround CIFS bug
          https://www.cyberciti.biz/tips/disable-caching-on-the-cifs-nfs-client.html

          wlad Vladislav Vaintroub added a comment - I suggest to try forcedirectio option for mounting, to workaround CIFS bug https://www.cyberciti.biz/tips/disable-caching-on-the-cifs-nfs-client.html

          Can't change mount option for the share, it's .... shared with other applications....
          But I understand this fails outside mariabackup.
          Could be useful to mention this incompatibility in the documentation maybe?

          rpizzi Rick Pizzi (Inactive) added a comment - Can't change mount option for the share, it's .... shared with other applications.... But I understand this fails outside mariabackup. Could be useful to mention this incompatibility in the documentation maybe?
          claudio.nanni Claudio Nanni added a comment -

          One case has been solved by mounting the CIFS with cache=none option.

          claudio.nanni Claudio Nanni added a comment - One case has been solved by mounting the CIFS with cache=none option.

          People

            wlad Vladislav Vaintroub
            rpizzi Rick Pizzi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.