Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13416

mariabackup --backup fails to copy log if LSN is above 4294967295

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.2.7
    • 10.2.8
    • Backup
    • None

    Description

      On a potentially corrupted server instance mariabackup fails with errno 14 EFAULT - Bad Address

      mariabackup --backup --user=root --password=... --target-dir=.../backup/$(date +\%Y-\%m-\%d_\%H:\%M:\%S) 
      170728 15:56:46 Connecting to MySQL server host: localhost, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
      Using server version 10.2.7-MariaDB-10.2.7+maria~xenial-log
      mariabackup based on MariaDB server 10.2.7-MariaDB Linux (x86_64)
      xtrabackup: uses posix_fadvise().
      xtrabackup: cd to /var/lib/mysql/
      xtrabackup: open files limit requested 215200, set to 215200
      xtrabackup: using the following InnoDB configuration:
      xtrabackup: innodb_data_home_dir = .
      xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup: innodb_log_group_home_dir = ./
      2017-07-28 15:56:46 140398601803584 [Note] InnoDB: Number of pools: 1
      mariabackup: Error writing file '.../backup/2017-07-28_15:56:46/ib_logfile0' (Errcode: 14 "Bad address")
      xtrabackup: Error: write to logfile failed
      xtrabackup: Error: xtrabackup_copy_logfile() failed.
      

      strace shows that after a few small reads from input file ib_logfile0 input file iblogfile1 is opened, 64KB are read from it, and an attempt to write ~18 ExaByte to the ib_logfile0 output file in the backup target dir is made.

      mkdir("/mysqldepot/backup/2017-07-29_16:47:46/", 0700) = 0
      open("./ib_logfile0", O_RDONLY)         = 4
      pread64(4, "\0\0\0\1\0\0\0\0\0\0\ft\370\36\200\0MariaDB 10.2.7\0\0"..., 512, 0) = 512
      pread64(4, "\0\0\0\0\0\4\373^\0\0\fx\237\1C\364\0\0\0\3\246\342\323\364\0\0\0\0\4\0\0\0"..., 512, 512) = 512
      pread64(4, "\0\0\0\0\0\4\373_\0\0\fx\237\1D\r\0\0\0\3\246\342\324\r\0\0\0\0\4\0\0\0"..., 512, 1536) = 512
      pread64(4, "\0\0\0\1\0\0\0\0\0\0\ft\370\36\200\0MariaDB 10.2.7\0\0"..., 512, 0) = 512
      pread64(4, "\0\0\0\0\0\4\373^\0\0\fx\237\1C\364\0\0\0\3\246\342\323\364\0\0\0\0\4\0\0\0"..., 512, 512) = 512
      pread64(4, "\0\0\0\0\0\4\373_\0\0\fx\237\1D\r\0\0\0\3\246\342\324\r\0\0\0\0\4\0\0\0"..., 512, 1536) = 512
      mkdir("/mysqldepot/backup/2017-07-29_16:47:46/", 0700) = -1 EEXIST (File exists)
      mkdir("/mysqldepot/backup/2017-07-29_16:47:46/", 0700) = -1 EEXIST (File exists)
      open("/mysqldepot/backup/2017-07-29_16:47:46/ib_logfile0", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 5
      write(5, "\0\0\0\1\0\0\0\0\0\0\fx\237\1D\rBackup 10.2.7-Ma"..., 512) = 512
      fadvise64(5, 0, 0, POSIX_FADV_DONTNEED) = 0
      write(5, "\0\0\0\0\0\4\373_\0\0\fx\237\1D\r\0\0\0\0\0\0\10\r\0\0\0\0\4\0\0\0"..., 512) = 512
      fadvise64(5, 0, 0, POSIX_FADV_DONTNEED) = 0
      write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
      fadvise64(5, 0, 0, POSIX_FADV_DONTNEED) = 0
      write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
      fadvise64(5, 0, 0, POSIX_FADV_DONTNEED) = 0
      open("./ib_logfile1", O_RDONLY)         = 6
      pread64(6, "\274O\200\243\0\26\0\r\0\4\373_\3648\0\0\fx\237\1D\r\0\0\0\0\0\0\0\0\0\0"..., 65536, 7094850560) = 65536
      write(5, "\274O\200\243\0\26\0\r\0\4\373_\3648\0\0\fx\237\1D\r\0\0\0\0\0\0\0\0\0\0"..., 18446730364173942784) = -1 EFAULT (Bad address)
      write(2, "\7", 1)                       = 1
      write(2, "mariabackup", 11)             = 11
      write(2, ": ", 2)                       = 2
      write(2, "Error writing file '/mysqldepot/"..., 99) = 99
      write(2, "\n", 1)                       = 1
      write(2, "xtrabackup: Error: write to logf"..., 43) = 43
      

      From looking at the error message text this seems to be happing in xtrabackup_scan_log_recs() in extra/mariabackup/xtrabackup.cc

      So either we have something overwriting the write_size variable there, giving it an insanely large value, or the values read and used to calculate the write_size are already corrupted to begin with.

      If the later is true the some sanity checks should be added to catch values that exceed innodb_log_file_size and report some more meaningful error message in that case ...

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              hholzgra Hartmut Holzgraefe
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.