[MXS-2611] MaxScale binlogrouter does not have a binlog index file Created: 2019-07-23  Updated: 2020-08-25  Resolved: 2020-05-25

Status: Closed
Project: MariaDB MaxScale
Component/s: binlogrouter
Affects Version/s: 2.3.9
Fix Version/s: 2.5.0

Type: New Feature Priority: Minor
Reporter: Manjot Singh (Inactive) Assignee: Todd Stoffel (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Epic Link: Pinloki Project

 Description   

Many binlog use cases rely on standards such as the index file. MaxScale does not create an index file on disk, it should.

The index file is a text file that lists the current binary log files.

In MariaDB Server, all binary log files share a common basename, which is defined by either the log_bin or log_basename options. To distinguish between log files, the basename is suffixed with sequential numbers using a .NNNNNN format. The basename can be queried at run-time by checking the log_bin_basename system variable.

https://mariadb.com/kb/en/library/replication-and-binary-log-system-variables/#log_bin

https://mariadb.com/kb/en/library/mysqld-options/#-log-basename

https://mariadb.com/kb/en/library/replication-and-binary-log-system-variables/#log_bin_basename

By default, the index file created by MariaDB Server shares the same basename as the binary log files, but it has a suffix of .index. This file name can be changed by defining the log_bin_index system variable.

https://mariadb.com/kb/en/library/replication-and-binary-log-system-variables/#log_bin_index

The default binary log file-naming basename is "HOSTNAME-bin". With the default basename, the binary log has files with names like this:

...
HOSTNAME-bin.0000101
HOSTNAME-bin.0000102
HOSTNAME-bin.0000103
...

Basically, the index file just needs to be written when a binary log is expired (remove the file from the top of the index) or when a new binary log is written (append to the index).

Additionally, an option should be created to allow setting permissions for all binlogs/index files, ie binlog_umask=0644

Possible binlogrouter use cases:

  • binlog forwarding from environments such as RDS where files are not accessible, for CDC type products such as GoldenGate and others.
  • PITR - binlog streaming to a central node running maxscale
  • hundreds of slaves, avoiding master context switching
  • moving replication through restricted firewalls, ie maxscale proxies from hundreds of database servers from one environment to slaves in another.
  • failover of slaves from one master to another, maxscale keeps a continuous connection so slaves never stop

Theoretically, you could put a maxscale on medium type of ec2 and have like 20 binlog routers with ports 4000-4020, each doing their own thing for different apps.



 Comments   
Comment by markus makela [ 2020-05-11 ]

The pinloki module currently has a binlog index file with absolute paths to binlogs.

Comment by markus makela [ 2020-05-25 ]

Pinloki implements a binlog index.

Generated at Thu Feb 08 04:15:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.