Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15779

mariabackup --apply-log-only fails if underlying file system is a CIFS mount

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.2.13
    • Fix Version/s: 10.1.33, 10.2.15
    • Component/s: Backup
    • Labels:
      None
    • Environment:
      Ubuntu

      Description

      When trying to restore a backup that was taken with same version of mariabackup, and that includes incremental pieces, the restore fails when we try to apply the first incremental
      to the base (full) backup.

      We found out that this is tied to the fact the underlying filesystem is a CIFS mount.
      The exact same operation with same data succeeds if we try this on a local filesystem.

      CIFS mount options:

      /Production-Backups/prod-mariadb03 on /backup type cifs (rw,relatime,vers=2.0,cache=strict,username=svc-backup-prod,domain=redacted,uid=0,noforceuid,gid=0,noforcegid,addr=10.15.200.31,file_mode=0775,dir_mode=0775,nounix,serverino,mapposix,rsize=65536,wsize=65536,echo_interval=60,actimeo=1)
      

      Command line used to apply incremental:

      $XTRABACKUP --defaults-file=$TD/backup-my.cnf --prepare --use-memory=4G --apply-log-only --target-dir=$TD --incremental-dir=$TD/incrementals/$piece 
      

      Failure log:

      mariabackup based on MariaDB server 10.2.13-MariaDB debian-linux-gnu (x86_64) 
      incremental backup from 1875122129 is enabled.
      mariabackup: cd to /backup/bgbackup-maria/rick/
      mariabackup: This target seems to be already prepared.
      mariabackup: using the following InnoDB configuration for recovery:
      mariabackup:   innodb_data_home_dir = .
      mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      mariabackup:   innodb_log_group_home_dir = /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01/
      mariabackup: Generating a list of tablespaces
      mariabackup: page size for /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta is 16384 bytes
      Applying /backup/bgbackup-maria/rick/incrementals/incr-2018-04-04_01-30-01//ibdata1.delta to ./ibdata1...
      2018-04-04 17:32:59 140111916681408 [Warning] InnoDB: Retry attempts for reading partial data failed.
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Tried to read 16384 bytes at offset 0, but was only able to read 0
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Operating system error number 22 in a file operation.
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
      2018-04-04 17:32:59 140111916681408 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
      2018-04-04 17:32:59 140111916681408 [ERROR] InnoDB: File (unknown): 'read' returned OS error 222. Cannot continue operation
      180404 17:32:59 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.2.13-MariaDB-10.2.13+maria~xenial
      key_buffer_size=0
      read_buffer_size=131072
      max_used_connections=0
      max_threads=1
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5421 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      addr2line: 'mariabackup': No such file
      mariabackup(my_print_stacktrace+0x2e)[0x55e29c08481e]
      mariabackup(handle_fatal_signal+0x345)[0x55e29bb81835]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6e58cc4390]
      linux/raise.c:54(__GI_raise)[0x7f6e56f6b428]
      stdlib/abort.c:91(__GI_abort)[0x7f6e56f6d02a]
      addr2line: 'mariabackup': No such file
      mariabackup(+0x9fd6b6)[0x55e29be906b6]
      mariabackup(+0xa030f8)[0x55e29be960f8]
      mariabackup(+0xa031b2)[0x55e29be961b2]
      mariabackup(+0x4324dc)[0x55e29b8c54dc]
      mariabackup(+0x408cb7)[0x55e29b89bcb7]
      mariabackup(+0x437053)[0x55e29b8ca053]
      mariabackup(main+0x185)[0x55e29b8ad145]
      csu/libc-start.c:325(__libc_start_main)[0x7f6e56f56830]
      addr2line: 'mariabackup': No such file
      mariabackup(_start+0x29)[0x55e29b8c1f79]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      

        Attachments

          Activity

            People

            Assignee:
            wlad Vladislav Vaintroub
            Reporter:
            rpizzi Rick Pizzi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration