Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-19492

Mariabackup hangs if table populated with INSERT... SELECT while it runs

Details

    • Bug
    • Status: Stalled (View Workflow)
    • Minor
    • Resolution: Unresolved
    • 10.2.24, 10.4.14
    • 10.4(EOL)
    • mariabackup
    • None

    Description

      Mariabackup consistently hangs when you load data into a table from another table while a full backup is running.

      It hangs in the following phase:

      [00] 2019-05-15 01:23:27 Waiting for log copy thread to read lsn 5607063606354

      Before that, the following errors appear:

      [00] 2019-05-15 00:25:44 Retrying read of log at LSN=5593738149376
      		 
      [00] 2019-05-15 00:25:44 Retrying read of log at LSN=5593738149376
      		 
      [00] 2019-05-15 00:25:44 Retrying read of log at LSN=5593738149376
      		 
      [00] 2019-05-15 00:25:44 Retrying read of log at LSN=5593738149376
      		 
      [00] 2019-05-15 00:25:44 Retrying read of log at LSN=5593738149376
      		 
      [00] 2019-05-15 00:25:44 xtrabackup_copy_logfile() failed.

      This is regularly happening at a customer of ours.
      I was able to reproduce in my lab as follows:

      1. start a full backup
      2. run insert into empytable select * from sbtest1;

      Command line used for backup:

      mariabackup --backup --user=root --password=xxx --parallel=4 --extra-lsndir=/tmp --stream=xbstream --target-dir=/tmp/2019-05-16/mariabackup

      Isolation level: default (REPEATABLE-READ)

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18611 mariabackup silently ended during xtrabackup_copy_logfile()

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31399 mariabackup fails with large innodb redo log - block is overwritten

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. MDEV-14992 BACKUP: in-server backup

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Task - A task that needs to be done. MDEV-23711 make mariabackup innodb redo log read error message more clear

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34062 mariadb-backup --backup is extremely slow at copying ib_logfile0

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-17029 mariabackup fails under load with schema changes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18612 mariabackup: Error: xtrabackup_copy_logfile() failed.

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 5 older comments
            vlad.lesin Vladislav Lesin added a comment -

            rpizzi yes it will go in 10.4 too. I hope it will be in the next 10.4 upcoming release. And yes, we can create special build for the customer. We need to know customer's platform and the version of MariaDB they use. But for 10.4 the steps to detect the problem, I mentioned above, must work without MDEV-23711 fix.

            I.e. if their backup log:

            (does not contain "Invalid log block checksum." or "xtrabackup_copy_logfile() failed: corrupt log.") and
            (innodb_log_checksums == 1 or redo log is encrypted),

            then this is indeed the duplicate of MDEV-18611.

            vlad.lesin Vladislav Lesin added a comment - rpizzi yes it will go in 10.4 too. I hope it will be in the next 10.4 upcoming release. And yes, we can create special build for the customer. We need to know customer's platform and the version of MariaDB they use. But for 10.4 the steps to detect the problem, I mentioned above, must work without MDEV-23711 fix. I.e. if their backup log: (does not contain "Invalid log block checksum." or "xtrabackup_copy_logfile() failed: corrupt log.") and (innodb_log_checksums == 1 or redo log is encrypted), then this is indeed the duplicate of MDEV-18611 .
            julien.fritsch Julien Fritsch added a comment -

            rpizzi please follow the instructions to create the needed Jira ticket and assign Vlad. https://docs.google.com/document/d/1flTIbF8vJQ7vB2FGHLiECo2DrzhUdgq4UgiKpqEXTjY/edit#heading=h.l24n0d50js1p

            julien.fritsch Julien Fritsch added a comment - rpizzi please follow the instructions to create the needed Jira ticket and assign Vlad. https://docs.google.com/document/d/1flTIbF8vJQ7vB2FGHLiECo2DrzhUdgq4UgiKpqEXTjY/edit#heading=h.l24n0d50js1p
            julien.fritsch Julien Fritsch added a comment -

            rpizzi?

            julien.fritsch Julien Fritsch added a comment - rpizzi ?
            marko Marko Mäkelä added a comment -

            In MDEV-14992 I wrote:

            MDEV-27621 is a case where backup cannot keep up with the server that is writing log. If the server process itself streamed the log like proposed in this task, it could trivially slow down the write workload as necessary.

            Making the server stream its ib_logfile0 to the backup should be much easier after the format change MDEV-14425 (MariaDB Server 10.8). That would implement only a subset of MDEV-14992; the full proposal there is that the server would generate an entire physical backup, which the InnoDB write-ahead log is only a part of.

            marko Marko Mäkelä added a comment - In MDEV-14992 I wrote: MDEV-27621 is a case where backup cannot keep up with the server that is writing log. If the server process itself streamed the log like proposed in this task, it could trivially slow down the write workload as necessary. Making the server stream its ib_logfile0 to the backup should be much easier after the format change MDEV-14425 (MariaDB Server 10.8). That would implement only a subset of MDEV-14992 ; the full proposal there is that the server would generate an entire physical backup, which the InnoDB write-ahead log is only a part of.
            marko Marko Mäkelä added a comment -

            MDEV-34062 could drastically improve this by enabling a memory-mapped interface to parsing the log.

            marko Marko Mäkelä added a comment - MDEV-34062 could drastically improve this by enabling a memory-mapped interface to parsing the log.

            People

              vlad.lesin Vladislav Lesin
              rpizzi Rick Pizzi (Inactive)
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.