[MDEV-33029] Add a mechanism to limit total binlog size on disk Created: 2023-12-14  Updated: 2023-12-18  Resolved: 2023-12-18

Status: Closed
Project: MariaDB Server
Component/s: Replication
Fix Version/s: N/A

Type: New Feature Priority: Major
Reporter: Hartmut Holzgraefe Assignee: Ralf Gebhardt
Resolution: Duplicate Votes: 1
Labels: None

Issue Links:
Duplicate
duplicates MDEV-31404 Implement binlog_space_limit In Testing

 Description   

Right now we only have an automatic purge mechanism for replication binary logs based on time (expire_log_days / binlog_expire_log_seconds), but not on total size, so there's a risk of running out of disk space due to binary logs piling up.

It is also not easy to determine the total space taken by binlog files as we only have SHOW BINLOGS, but no INFORMATION_SCHEMA or PERFORMANCE_SCHEMA table providing the same information, so that it's not possible to apply aggregate functions like e.g.:

SELECT SUM(binlog_file_size) AS total_size FROM I_S.BINLOG_FILES;

which could be used from within a scheduled EVENT to issue PURGE BINLOG commands in case a certain size threshold is exceeded.

Adding a new setting like binlog_space_limit, after which the oldest binlog file would automatically be purged, would be nice as it would allow for more reliable file system capacity / quota planning,

We already have a similar relay_log_space_limit, but in case of the binlog(s) there should be a purge instead of a write stall.



 Comments   
Comment by Kristian Nielsen [ 2023-12-14 ]

Monty very recently implemented this in MDEV-31404.

Good point about the similarly named --relay-log-space-limit, it might be more consistent to make --binlog-space-limit be the only name for the option (now it's an alias for --max-binlog-total-size).

The MDEV-31404 patch also provides the total space used by binlogs as show status like "binlog_disk_use";

Generated at Thu Feb 08 10:35:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.