Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14536

backup fails after "log block has not been fully written by server, will retry later"

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1(EOL), 10.2(EOL)
    • 10.1.30, 10.2.12
    • Backup
    • None

    Description

      Backup can fail after the message "log block has not been fully written by the server, will retry later" , for example on log block checksum mistmatch. Indeed, the backup reads from disk, while server writes to disk, and the writes are not atomi, so mismatch in checksum can happen.

      However, backup does not re-read from disk, and the backup fails in this case

      171120 23:46:30 [01] Copying ./mysql/innodb_table_stats.frm to /CIBackup/dbbackup/TestMbk/mysql/innodb_table_stats.frm
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [01] Copying ./mysql/column_stats.MYD to /CIBackup/dbbackup/TestMbk/mysql/column_stats.MYD
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [01] Copying ./mysql/proxies_priv.MYI to /CIBackup/dbbackup/TestMbk/mysql/proxies_priv.MYI
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [01] Copying ./mysql/general_log.CSM to /CIBackup/dbbackup/TestMbk/mysql/general_log.CSM
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [01] Copying ./mysql/time_zone_transition.MYD to /CIBackup/dbbackup/TestMbk/mysql/time_zone_transition.MYD
      171120 23:46:30 [01]        ...done
      171120 23:46:30 Finished backing up non-InnoDB tables and files
      171120 23:46:30 [01] Copying aria_log_control to /CIBackup/dbbackup/TestMbk/aria_log_control
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [01] Copying aria_log.00000001 to /CIBackup/dbbackup/TestMbk/aria_log.00000001
      171120 23:46:30 [01]        ...done
      171120 23:46:30 [00] Writing xtrabackup_binlog_info
      171120 23:46:30 [00]        ...done
      171120 23:46:30 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
      mariabackup: The latest check point (for incremental): '12752751793617'
      mariabackup: Stopping log copying thread.
      .mariabackup: warning: Log block checksum mismatch (block no 211656395 at lsn 12752751793152):
      expected 595842909, calculated checksum 1480124372
      mariabackup: warning: this is possible when the log block has not been fully written by the server, will retry later.
      171120 23:46:30 >> log scanned up to (12752751793152)
       
      171120 23:46:30 Executing UNLOCK TABLES
      171120 23:46:30 All tables unlocked
      171120 23:46:30 Backup created in directory '/CIBackup/dbbackup/TestMbk/'
      MySQL binlog position: filename 'mariadb-bin.000080', position '406', GTID of the last change '0-7725-126,11-7511-6574751,15-7717-2768443781,17-7726-2269635431,21-7521-7596336,25-125-1409128945,27-171-802844348,35-7735-170882928,55-155-2834379'
      171120 23:46:30 [00] Writing backup-my.cnf
      171120 23:46:30 [00]        ...done
      171120 23:46:30 [00] Writing xtrabackup_info
      171120 23:46:30 [00]        ...done
      mariabackup: Transaction log of lsn (12752751793617) to (12752751793152) was copied.
      mariabackup: error: last checkpoint LSN (12752751793617) is larger than last copied LSN (12752751793152).
      

      Attachments

        Activity

          wlad Vladislav Vaintroub created issue -
          wlad Vladislav Vaintroub made changes -
          Field Original Value New Value
          Assignee Vladislav Vaintroub [ wlad ]
          wlad Vladislav Vaintroub made changes -
          Description Backup can fail after the message "log block has not been fully written by the server, will retry later" , for example on log block checksum mistmatch. Indeed, the backup reads from disk, while server writes to disk, and the writes are not atomi, so mismatch in checksum can happen.

          However, backup does not re-read from disk, and the backup fails in this case
          Backup can fail after the message "log block has not been fully written by the server, will retry later" , for example on log block checksum mistmatch. Indeed, the backup reads from disk, while server writes to disk, and the writes are not atomi, so mismatch in checksum can happen.

          However, backup does not re-read from disk, and the backup fails in this case

          {noformat}
          171120 23:46:30 [01] Copying ./mysql/innodb_table_stats.frm to /CIBackup/dbbackup/TestMbk/mysql/innodb_table_stats.frm
          171120 23:46:30 [01] ...done
          171120 23:46:30 [01] Copying ./mysql/column_stats.MYD to /CIBackup/dbbackup/TestMbk/mysql/column_stats.MYD
          171120 23:46:30 [01] ...done
          171120 23:46:30 [01] Copying ./mysql/proxies_priv.MYI to /CIBackup/dbbackup/TestMbk/mysql/proxies_priv.MYI
          171120 23:46:30 [01] ...done
          171120 23:46:30 [01] Copying ./mysql/general_log.CSM to /CIBackup/dbbackup/TestMbk/mysql/general_log.CSM
          171120 23:46:30 [01] ...done
          171120 23:46:30 [01] Copying ./mysql/time_zone_transition.MYD to /CIBackup/dbbackup/TestMbk/mysql/time_zone_transition.MYD
          171120 23:46:30 [01] ...done
          171120 23:46:30 Finished backing up non-InnoDB tables and files
          171120 23:46:30 [01] Copying aria_log_control to /CIBackup/dbbackup/TestMbk/aria_log_control
          171120 23:46:30 [01] ...done
          171120 23:46:30 [01] Copying aria_log.00000001 to /CIBackup/dbbackup/TestMbk/aria_log.00000001
          171120 23:46:30 [01] ...done
          171120 23:46:30 [00] Writing xtrabackup_binlog_info
          171120 23:46:30 [00] ...done
          171120 23:46:30 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
          mariabackup: The latest check point (for incremental): '12752751793617'
          mariabackup: Stopping log copying thread.
          .mariabackup: warning: Log block checksum mismatch (block no 211656395 at lsn 12752751793152):
          expected 595842909, calculated checksum 1480124372
          mariabackup: warning: this is possible when the log block has not been fully written by the server, will retry later.
          171120 23:46:30 >> log scanned up to (12752751793152)

          171120 23:46:30 Executing UNLOCK TABLES
          171120 23:46:30 All tables unlocked
          171120 23:46:30 Backup created in directory '/CIBackup/dbbackup/TestMbk/'
          MySQL binlog position: filename 'mariadb-bin.000080', position '406', GTID of the last change '0-7725-126,11-7511-6574751,15-7717-2768443781,17-7726-2269635431,21-7521-7596336,25-125-1409128945,27-171-802844348,35-7735-170882928,55-155-2834379'
          171120 23:46:30 [00] Writing backup-my.cnf
          171120 23:46:30 [00] ...done
          171120 23:46:30 [00] Writing xtrabackup_info
          171120 23:46:30 [00] ...done
          mariabackup: Transaction log of lsn (12752751793617) to (12752751793152) was copied.
          mariabackup: error: last checkpoint LSN (12752751793617) is larger than last copied LSN (12752751793152).
          {noformat}
          wlad Vladislav Vaintroub made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          wlad Vladislav Vaintroub made changes -
          issue.field.resolutiondate 2017-11-30 16:32:31.0 2017-11-30 16:32:31.846
          wlad Vladislav Vaintroub made changes -
          Fix Version/s 10.1.30 [ 22637 ]
          Fix Version/s 10.2.12 [ 22810 ]
          Fix Version/s 10.2 [ 14601 ]
          Fix Version/s 10.1 [ 16100 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 84206 ] MariaDB v4 [ 153270 ]

          People

            wlad Vladislav Vaintroub
            wlad Vladislav Vaintroub
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.