Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17029

mariabackup fails under load with schema changes

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.2.16
    • N/A
    • mariabackup
    • None
    • Ubuntu 18:04, Ubuntu 16:04, Galera cluster

    Description

      When we run mariabackup on a galera cluster nodes with a lot of data/databases (over 1500 databases with 700GB of data at the moment), if the servers are under moderate use mariabackup tool fails with the following log messages:

      Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: 2018-08-21 17:44:51 140002715002816 [Note] InnoDB: Read redo log up to LSN=1691316337664
      		 
      Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: mariabackup: Error: xtrabackup_copy_logfile() failed.
      		 
      Aug 21 17:44:52 maria3 -wsrep-sst-donor[25825]: mariabackup finished with error: 1.  Check /data/maria//innobackup.backup.log
      		 
      Aug 21 17:44:52 maria3 -wsrep-sst-donor[25826]: Cleanup after exit with status:22

      The cluster is under load, but I think these are happening when schema is being modified (databases created, dropped and so on). It could be related to tables flooded with records, because when we create databases we populate them with a lot of records.

      This happens both when running mariabackup manually or when it's triggered by SST.

      The log file mentioned in the message "innobackup.backup.log" cannot be found anywhere on the system. So we have no more info about the error.

      Unfortunately, this is a production cluster and we can't do much in a way of debugging on these machines.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19492 Mariabackup hangs if table populated with INSERT... SELECT while it runs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Stalled

          Task - A task that needs to be done. MDEV-16791 mariabackup : allow consistent backup, in presence of concurrent DDL, also without --lock-ddl-per-table

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18611 mariabackup silently ended during xtrabackup_copy_logfile()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18612 mariabackup: Error: xtrabackup_copy_logfile() failed.

          • Major - Major loss of function.
          • Closed

          Activity

            wlad Vladislav Vaintroub added a comment -

            I'm going to close it since there is no actionable info about the bug.
            The description is vaguely similar to https://jira.mariadb.org/browse/MDEV-16791 . You can reopen (and provide more information, incl the full log of mariabackup run), if it happens after 10.2.18

            wlad Vladislav Vaintroub added a comment - I'm going to close it since there is no actionable info about the bug. The description is vaguely similar to https://jira.mariadb.org/browse/MDEV-16791 . You can reopen (and provide more information, incl the full log of mariabackup run), if it happens after 10.2.18

            People

              wlad Vladislav Vaintroub
              kvasserman Konstantin Vasserman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.