Binlog compression and archiving

This Jira was originally "Stream binary logs to s3 bucket".
It has been implemented with two new features: compression and archiving.
See https://jira.mariadb.org/browse/MXS-4867.

Original Description:

Hi Team,

Need to stream the binary logs directly to S3 bucket without storing locally.

This is new feature request please let me know your thoughts.

I have tested this in my test environment like below,

Step 1 :

Created new S3 bucket and provided all access to that bucket.

Step 2 :

Installed the maxscale 6 and configured binlog router.

Step 3 :

Then locally mounted that S3 bucket on my maxscale server.

Example :

[root@centos14 /]# df -h | grep bucket

s3fs                     256T     0  256T   0% /mariadb-s3-bucket

Step 4 :

Added this mount path in my maxscale config.

Example :

[replication-service]

type=service

router=binlogrouter

cluster=MariaDB-Monitor

select_master=true

expire_log_minimum_files=10000

user=xxx

password=xxxx

datadir=/mariadb-s3-bucket/binlogs/

Step 5:

Start the maxscale service. Now binlogs are stored to S3 bucket.

[root@centos14 binlogs]# pwd

/mariadb-s3-bucket/binlogs

[root@centos14 binlogs]# 

[root@centos14 binlogs]# ls -lrth

total 4.0K

-rw-r--r--. 1 maxscale maxscale   12 Jan 23 08:03 requested_rpl_state

-rw-r--r--. 1 maxscale maxscale 1.8K Jan 23 08:03 centos11-bin.000001

-rw-r--r--. 1 maxscale maxscale   48 Jan 23 08:03 binlog.index

-rw-r--r--. 1 maxscale maxscale   12 Jan 23 08:08 rpl_state

-rw-r--r--. 1 maxscale maxscale  258 Jan 23 11:32 master-info.json

Problem statement 1 :

The streaming speed is too slow compared to local storage. I loaded some data using sysbench but it's taking more time stream to S3 bucket. The transfer speed is 4MB and it's streaming single threaded.

Is it possible to stream the binlogs without mounting locally ? or how can we speed up this process using binlog router.

Problem statement 2 :

If unexpectedly my maxscale crashes it removes existing binlogs from S3 bucket.