Provide the possibility to bypass filesystem buffering for InnoDB-based binlogs

As a DBA, the thing I really hate is not having control over the resource consumption of my systems. No real control means no predictable values and eventually issues arising without knowing the reason immediately.

In Linux, there is something that annoys me a lot, and this is not MariaDB-related only: the filesystem cache.

Even if there are some kernel parameters to try to limit that, by experience, this doesn't really work all the time.

I've seen a lot of systems with very low swappiness that still swap anyway, even with plenty of RAM available, because the filesystem cache is using it!

That's why I've always been a big fan of O_DIRECT. However, for some logs, there is no way to bypass the FS buffering. A long time ago, Percona Server had an ALL_O_DIRECT value for the innodb_flush_method variable.

The binlogs are among the logs where access is made using the FS cache. So if you add a new replica that needs to process a lot of binlogs, or if you run show binlog events, or you use mysqlbinlog for any reason, you can end up with a large amount of RAM used by the FS cache because of the binlogs loaded into it.

Regarding performance, for replicas in sync, this should not make any difference, as they use the FIFO page cache. Now, if multiple replicas need to read many binlogs, that might indeed affect the performance a bit, certainly on slow systems.