Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33029

Add a mechanism to limit total binlog size on disk

Details

    • New Feature
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • N/A
    • Replication
    • None

    Description

      Right now we only have an automatic purge mechanism for replication binary logs based on time (expire_log_days / binlog_expire_log_seconds), but not on total size, so there's a risk of running out of disk space due to binary logs piling up.

      It is also not easy to determine the total space taken by binlog files as we only have SHOW BINLOGS, but no INFORMATION_SCHEMA or PERFORMANCE_SCHEMA table providing the same information, so that it's not possible to apply aggregate functions like e.g.:

      SELECT SUM(binlog_file_size) AS total_size FROM I_S.BINLOG_FILES;

      which could be used from within a scheduled EVENT to issue PURGE BINLOG commands in case a certain size threshold is exceeded.

      Adding a new setting like binlog_space_limit, after which the oldest binlog file would automatically be purged, would be nice as it would allow for more reliable file system capacity / quota planning,

      We already have a similar relay_log_space_limit, but in case of the binlog(s) there should be a purge instead of a write stall.

      Attachments

        Issue Links

          Activity

            Monty very recently implemented this in MDEV-31404.

            Good point about the similarly named --relay-log-space-limit, it might be more consistent to make --binlog-space-limit be the only name for the option (now it's an alias for --max-binlog-total-size).

            The MDEV-31404 patch also provides the total space used by binlogs as show status like "binlog_disk_use";

            knielsen Kristian Nielsen added a comment - Monty very recently implemented this in MDEV-31404 . Good point about the similarly named --relay-log-space-limit, it might be more consistent to make --binlog-space-limit be the only name for the option (now it's an alias for --max-binlog-total-size). The MDEV-31404 patch also provides the total space used by binlogs as show status like "binlog_disk_use";

            I would like to point out a major difference between --binlog-space-limit and --relay-log-space-limit. The relay log limit does not automatically delete files when it is hit. I believe that it should.

            If --relay-log-purge is set (the default), then the relay logs are deleted as soon as they are consumed. If it is not set, then the relay logs will expand up to --relay-log-space-limit, but (unless something has changed?) they are NOT automatically deleted at that point.

            I believe that the relay logs should be deleted in that situation if they have been consumed. This would align the --relay-log-space-limit behavior with that of --binlog-space-limit. Really without this behavior, the ability to disable --relay-log-purge isn't very useful.

            Please see MDEV-29195.

            xan@biblionix.com Xan Charbonnet added a comment - I would like to point out a major difference between --binlog-space-limit and --relay-log-space-limit. The relay log limit does not automatically delete files when it is hit. I believe that it should. If --relay-log-purge is set (the default), then the relay logs are deleted as soon as they are consumed. If it is not set, then the relay logs will expand up to --relay-log-space-limit, but (unless something has changed?) they are NOT automatically deleted at that point. I believe that the relay logs should be deleted in that situation if they have been consumed. This would align the --relay-log-space-limit behavior with that of --binlog-space-limit. Really without this behavior, the ability to disable --relay-log-purge isn't very useful. Please see MDEV-29195 .

            People

              ralf.gebhardt Ralf Gebhardt
              hholzgra Hartmut Holzgraefe
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.