Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28758

Mariabackup copies binary logs to backup directory

Details

    Description

      Hi,

      I have log_bin specified to the data directory:
      /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

      In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
      [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

      This is needless consumption of space on the backup disk.

      I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

      Specifying a custom binlog directory is NOT a work-around for this issue. It does not seam that the general log is stored with the backup, so why can the binary logs not be excepted?

      Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup.

      My current version:
      mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
      [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
      /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

      The older runs were slightly different versions:
      [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
      /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

      Thank you,
      Michaƫl

      Attachments

        Issue Links

          Activity

            michaeldg Michaƫl de groot created issue -
            michaeldg Michaƫl de groot made changes -
            Field Original Value New Value
            Description Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            As a work-around I will have a customer directory for binary logs, but I do think it is worth to fix this bug. What if the same bug applies to slow logs, or the general log after it was running for a few days?

            If those other logs are excepted, then why can the binary logs not be excepted?

            Thank you,
            Michaƫl
            Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            As a work-around I will have a customer directory for binary logs, but I do think it is worth to fix this bug. What if the same bug applies to slow logs, or the general log after it was running for a few days?

            If those other logs are excepted, then why can the binary logs not be excepted?

            Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup. In the

            My current version:
            mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
            [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

            The older runs were slightly different versions:
            [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

            Thank you,
            Michaƫl
            danblack Daniel Black added a comment -

            Is this not covered with --binlog-info=off?

            danblack Daniel Black added a comment - Is this not covered with --binlog-info=off ?

            According to the documentation, this is to retrieve the binary log file and position. Not the binary logs itself.

            michaeldg Michaƫl de groot added a comment - According to the documentation, this is to retrieve the binary log file and position. Not the binary logs itself.
            michaeldg Michaƫl de groot made changes -
            Priority Minor [ 4 ] Major [ 3 ]

            The same thing happens from the mariabackup SST script. I have therefore increased the priority.

            michaeldg Michaƫl de groot added a comment - The same thing happens from the mariabackup SST script. I have therefore increased the priority.
            emaijala Ere Maijala added a comment -

            I believe this is the reason we are now experiencing downtime due to the read lock being held while the large binary logs are being copied. With previous mariabackup versions the read lock was held for a few seconds, but now it's 44 minutes.

            emaijala Ere Maijala added a comment - I believe this is the reason we are now experiencing downtime due to the read lock being held while the large binary logs are being copied. With previous mariabackup versions the read lock was held for a few seconds, but now it's 44 minutes.
            juan.vera Juan made changes -
            Affects Version/s 10.3.35 [ 27512 ]
            Environment RHEL

            We found a working workaround for a customer with the same issue. We replaced the mariabackup binary containing the bug (version 10.3.35), with the last previously working binary (version 10.3.34).

            1) backup the old binary: sudo cp /usr/bin/mariabackup /usr/bin/mariabackup_35
            2) copy or install mariabackup version 10.3.34 to: /usr/bin/mariabackup

            martin.reinhardt@mariadb.com Martin Reinhardt added a comment - We found a working workaround for a customer with the same issue. We replaced the mariabackup binary containing the bug (version 10.3.35), with the last previously working binary (version 10.3.34). 1) backup the old binary: sudo cp /usr/bin/mariabackup /usr/bin/mariabackup_35 2) copy or install mariabackup version 10.3.34 to: /usr/bin/mariabackup
            psumner Phil Sumner added a comment - - edited

            I'm experiencing what I suspect is the same issue in 10.6.8

            I recently upgraded a replica used exclusively for taking backups from 10.1.48 -> 10.6.8, and using mostly the same command line for mariabackup instead of innobackupex - backup times have at least doubled and backup sizes have gone from 25-30GB to 120GB, and replication comes to a grinding halt while the BACKUP LOCK is taken and held while 2 days of binary logs are also backed up.

            Old:
            innobackupex --parallel=4 --compress --stream=xbstream ./ 2> /var/log/backup.log | ssh admin@host"cat > *path*/$(date +'%Y%m%d').xbstream

            New:
            mariabackup --backup --parallel=4 --stream=xbstream | pigz -p4 -c | ssh admin@host "cat > *path*/$(date +'%Y%m%d').xbstream.gz

            I reverted Maria Backup to 10.6.7 and the behaviour is now as expected - i.e. binlogs are NOT copied. Backup is still slower and larger than in 10.1 with Xtrabackup but it's better than it was.

            psumner Phil Sumner added a comment - - edited I'm experiencing what I suspect is the same issue in 10.6.8 I recently upgraded a replica used exclusively for taking backups from 10.1.48 -> 10.6.8, and using mostly the same command line for mariabackup instead of innobackupex - backup times have at least doubled and backup sizes have gone from 25-30GB to 120GB, and replication comes to a grinding halt while the BACKUP LOCK is taken and held while 2 days of binary logs are also backed up. Old: innobackupex --parallel=4 --compress --stream=xbstream ./ 2> /var/log/backup.log | ssh admin@host"cat > * path */$(date +'%Y%m%d').xbstream New: mariabackup --backup --parallel=4 --stream=xbstream | pigz -p4 -c | ssh admin@host "cat > * path */$(date +'%Y%m%d').xbstream.gz I reverted Maria Backup to 10.6.7 and the behaviour is now as expected - i.e. binlogs are NOT copied. Backup is still slower and larger than in 10.1 with Xtrabackup but it's better than it was.
            antonp1976 Anton Petin added a comment - - edited

            Same bug in mariabackup 10.5.16, there is no such problem in 10.5.15 with same cmd

            mariabackup --user=root --password=xxx  --target-dir /home/db/b/tmp/$date --backup  --slave-info --safe-slave-backup --parallel=10
            

            antonp1976 Anton Petin added a comment - - edited Same bug in mariabackup 10.5.16, there is no such problem in 10.5.15 with same cmd mariabackup --user=root --password=xxx --target-dir /home/db/b/tmp/$date --backup --slave-info --safe-slave-backup --parallel= 10
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.3 [ 22126 ]
            julien.fritsch Julien Fritsch made changes -
            Assignee Alexander Barkov [ bar ]
            michaeldg Michaƫl de groot made changes -
            Affects Version/s 10.6.8 [ 27506 ]
            Affects Version/s 10.5.16 [ 27508 ]
            michaeldg Michaƫl de groot made changes -
            Description Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            As a work-around I will have a customer directory for binary logs, but I do think it is worth to fix this bug. What if the same bug applies to slow logs, or the general log after it was running for a few days?

            If those other logs are excepted, then why can the binary logs not be excepted?

            Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup. In the

            My current version:
            mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
            [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

            The older runs were slightly different versions:
            [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

            Thank you,
            Michaƫl
            Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            As a work-around I will have a custom directory for binary logs, but I do think it is worth to fix this bug. What if the same bug applies to slow logs, or the general log after it was running for a few days?

            If those other logs are excepted, then why can the binary logs not be excepted?

            Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup. In the

            My current version:
            mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
            [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

            The older runs were slightly different versions:
            [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

            Thank you,
            Michaƫl
            StefanMay Stefan May added a comment -

            I'm also seeing the issue in version 10.5.16 on openSuse Leap 15.3 x86_64

            Im my configuration, all binlog files are stored in the main database directory /var/lib/mysql/
            Version 10.5.16 copies all binlog files into the backup directory, significantly increasing lock duration and backup size.
            The previous version 10.5.15 did not include binlog files into the backup directory.

            In my particular server configuration with automated backups stored on a remote host, these bloated backups impact server performance and waste resources (network traffic and backup space).
            As a temporary worakround, I now use a low value for server variable "expire_logs_days" (alternatively "binlog_expire_logs_seconds") to purge binary logs faster than previously configured.

            StefanMay Stefan May added a comment - I'm also seeing the issue in version 10.5.16 on openSuse Leap 15.3 x86_64 Im my configuration, all binlog files are stored in the main database directory /var/lib/mysql/ Version 10.5.16 copies all binlog files into the backup directory, significantly increasing lock duration and backup size. The previous version 10.5.15 did not include binlog files into the backup directory. In my particular server configuration with automated backups stored on a remote host, these bloated backups impact server performance and waste resources (network traffic and backup space). As a temporary worakround, I now use a low value for server variable "expire_logs_days" (alternatively "binlog_expire_logs_seconds") to purge binary logs faster than previously configured.
            michaeldg Michaƫl de groot made changes -
            Description Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            As a work-around I will have a custom directory for binary logs, but I do think it is worth to fix this bug. What if the same bug applies to slow logs, or the general log after it was running for a few days?

            If those other logs are excepted, then why can the binary logs not be excepted?

            Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup. In the

            My current version:
            mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
            [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

            The older runs were slightly different versions:
            [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

            Thank you,
            Michaƫl
            Hi,

            I have log_bin specified to the data directory:
            /etc/my.cnf.d/common.cnf:12:log_bin="/var/lib/mysql/mariadb"

            In the mariabackup output, I see that mariabackup copies the binary logs to the backup directory:
            [00] 2022-06-06 19:22:58 Compressing /var/lib/mysql/mariadb.000270 to /backupdisk/diff-2022-06-06_19-07-01/mariadb.000270.qp

            This is needless consumption of space on the backup disk.

            I did not find a command-line switch to disable copying the binary logs, I think this is a bug in mariabackup to just copy all the files that are in the data directory.

            Specifying a custom binlog directory is NOT a work-around for this issue. It does not seam that the general log is stored with the backup, so why can the binary logs not be excepted?

            Earlier bakup runs (with a different mariabackup version) did not show this issue, it seams some thing changed in mariabackup.

            My current version:
            mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)
            [00] 2022-06-06 10:26:07 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.35-MariaDB Linux (x86_64)

            The older runs were slightly different versions:
            [00] 2022-03-23 00:07:01 Using server version 10.3.31-MariaDB-log
            /usr/bin/mariabackup based on MariaDB server 10.3.34-MariaDB Linux (x86_64)

            Thank you,
            Michaƫl

            I have tried to implement the work-around of binary logs in a separate directory. This unfortunately did not make a difference. The binary logs are still copied from the custom binlog directory in the backup location.

            michaeldg Michaƫl de groot added a comment - I have tried to implement the work-around of binary logs in a separate directory. This unfortunately did not make a difference. The binary logs are still copied from the custom binlog directory in the backup location.
            antonp1976 Anton Petin added a comment -

            This is a very serious bug, cause we store bin logs and snapshots of data for a long time. Each snapshot is around 1Tb and binlogs for 1Tb at all, so when we take an snapshot with new version of mariabackup - full backup is around 2Tb and long lock time while backuping.

            Now we are using mariabackup 10.5.15 on 10.5.16 server, but i think it's not normal for such company like MariaDB not to solve it.

            antonp1976 Anton Petin added a comment - This is a very serious bug, cause we store bin logs and snapshots of data for a long time. Each snapshot is around 1Tb and binlogs for 1Tb at all, so when we take an snapshot with new version of mariabackup - full backup is around 2Tb and long lock time while backuping. Now we are using mariabackup 10.5.15 on 10.5.16 server, but i think it's not normal for such company like MariaDB not to solve it.
            ccalender Chris Calender (Inactive) made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            marko Marko MƤkelƤ made changes -
            ralf.gebhardt Ralf Gebhardt made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            serg Sergei Golubchik made changes -
            Assignee Alexander Barkov [ bar ] Julius Goryavsky [ sysprg ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Julius Goryavsky [ sysprg ] Sergei Golubchik [ serg ]
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            Status Open [ 1 ] Confirmed [ 10101 ]
            ralf.gebhardt Ralf Gebhardt made changes -
            Status Confirmed [ 10101 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            serg Sergei Golubchik made changes -
            Affects Version/s 10.4.25 [ 27510 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Julius Goryavsky [ sysprg ]
            serg Sergei Golubchik made changes -
            sysprg Julius Goryavsky made changes -
            Assignee Julius Goryavsky [ sysprg ] Jan Lindstrƶm [ jplindst ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            sysprg Julius Goryavsky made changes -

            After an Update we also ran into this Bug. Please fix this or give us a hint how to prevent the creation of binlog archives in backup dir.

            mariadb-backup --backup --host=localhost --user=root --password=xxx--target-dir /export/mariadbbackup/$(hostname)/$date --parallel=8 --compress --compress-threads=4
            

            avalor Andreas Hering added a comment - After an Update we also ran into this Bug. Please fix this or give us a hint how to prevent the creation of binlog archives in backup dir. mariadb-backup --backup --host=localhost --user=root --password=xxx--target-dir /export/mariadbbackup/$(hostname)/$date --parallel=8 --compress --compress-threads=4
            ralf.gebhardt Ralf Gebhardt made changes -
            Assignee Jan Lindstrƶm [ jplindst ] Sergei Golubchik [ serg ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Jan Lindstrƶm [ jplindst ]
            serg Sergei Golubchik added a comment - - edited

            be286c57a looks ok to me

            serg Sergei Golubchik added a comment - - edited be286c57a looks ok to me

            ok to push

            jplindst Jan Lindstrƶm (Inactive) added a comment - ok to push
            jplindst Jan Lindstrƶm (Inactive) made changes -
            Assignee Jan Lindstrƶm [ jplindst ] Julius Goryavsky [ sysprg ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sysprg Julius Goryavsky added a comment - https://github.com/MariaDB/server/commit/7fb1f919d0629d0c2bc68c616de557b658068fa0 https://github.com/MariaDB/server/commit/5b4154373a93b323c3502692c46dcc51b1731785 https://github.com/MariaDB/server/commit/5197519f4f7b31c61e3abccf75650e5356aecd4c
            sysprg Julius Goryavsky made changes -
            issue.field.resolutiondate 2022-08-01 16:57:01.0 2022-08-01 16:57:01.433
            sysprg Julius Goryavsky made changes -
            Component/s Galera SST [ 10121 ]
            Component/s mariabackup [ 14500 ]
            Fix Version/s 10.3.36 [ 27513 ]
            Fix Version/s 10.4.26 [ 27511 ]
            Fix Version/s 10.5.17 [ 27509 ]
            Fix Version/s 10.6.9 [ 27507 ]
            Fix Version/s 10.7.5 [ 27505 ]
            Fix Version/s 10.8.4 [ 27503 ]
            Fix Version/s 10.9.2 [ 27115 ]
            Fix Version/s 10.10.0 [ 27912 ]
            Fix Version/s 10.3 [ 22126 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            sysprg Julius Goryavsky made changes -
            Richard Richard Stracke made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 116776 172346 126233

            People

              sysprg Julius Goryavsky
              michaeldg Michaƫl de groot
              Votes:
              13 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.