Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17029

mariabackup fails under load with schema changes

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Incomplete
    • Affects Version/s: 10.2.16
    • Fix Version/s: N/A
    • Component/s: mariabackup
    • Labels:
      None
    • Environment:
      Ubuntu 18:04, Ubuntu 16:04, Galera cluster

      Description

      When we run mariabackup on a galera cluster nodes with a lot of data/databases (over 1500 databases with 700GB of data at the moment), if the servers are under moderate use mariabackup tool fails with the following log messages:

      Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: 2018-08-21 17:44:51 140002715002816 [Note] InnoDB: Read redo log up to LSN=1691316337664
      Aug 21 17:44:51 maria3 -innobackupex-backup[25752]: mariabackup: Error: xtrabackup_copy_logfile() failed.
      Aug 21 17:44:52 maria3 -wsrep-sst-donor[25825]: mariabackup finished with error: 1.  Check /data/maria//innobackup.backup.log
      Aug 21 17:44:52 maria3 -wsrep-sst-donor[25826]: Cleanup after exit with status:22
      

      The cluster is under load, but I think these are happening when schema is being modified (databases created, dropped and so on). It could be related to tables flooded with records, because when we create databases we populate them with a lot of records.

      This happens both when running mariabackup manually or when it's triggered by SST.

      The log file mentioned in the message "innobackup.backup.log" cannot be found anywhere on the system. So we have no more info about the error.

      Unfortunately, this is a production cluster and we can't do much in a way of debugging on these machines.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wlad Vladislav Vaintroub
                Reporter:
                kvasserman Konstantin Vasserman
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: