Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-3986

Stream binary logs to s3 bucket

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 22.08
    • Fix Version/s: Icebox
    • Component/s: binlogrouter
    • Labels:
    • Environment:
      All Linux Environments

      Description

      Hi Team,

      Need to stream the binary logs directly to S3 bucket without storing locally.

      This is new feature request please let me know your thoughts.

      I have tested this in my test environment like below,

      Step 1 :

      Created new S3 bucket and provided all access to that bucket.

      Step 2 :

      Installed the maxscale 6 and configured binlog router.

      Step 3 :

      Then locally mounted that S3 bucket on my maxscale server.

      Example :

      [root@centos14 /]# df -h | grep bucket
      s3fs                     256T     0  256T   0% /mariadb-s3-bucket
      
      

      Step 4 :

      Added this mount path in my maxscale config.

      Example :

      [replication-service]
      type=service
      router=binlogrouter
      cluster=MariaDB-Monitor
      select_master=true
      expire_log_minimum_files=10000
      user=xxx
      password=xxxx
      datadir=/mariadb-s3-bucket/binlogs/
      

      Step 5:

      Start the maxscale service. Now binlogs are stored to S3 bucket.

      [root@centos14 binlogs]# pwd
      /mariadb-s3-bucket/binlogs
      [root@centos14 binlogs]# 
      [root@centos14 binlogs]# ls -lrth
      total 4.0K
      -rw-r--r--. 1 maxscale maxscale   12 Jan 23 08:03 requested_rpl_state
      -rw-r--r--. 1 maxscale maxscale 1.8K Jan 23 08:03 centos11-bin.000001
      -rw-r--r--. 1 maxscale maxscale   48 Jan 23 08:03 binlog.index
      -rw-r--r--. 1 maxscale maxscale   12 Jan 23 08:08 rpl_state
      -rw-r--r--. 1 maxscale maxscale  258 Jan 23 11:32 master-info.json
      

      Problem statement 1 :

      The streaming speed is too slow compared to local storage. I loaded some data using sysbench but it's taking more time stream to S3 bucket. The transfer speed is 4MB and it's streaming single threaded.

      Is it possible to stream the binlogs without mounting locally ? or how can we speed up this process using binlog router.

      Problem statement 2 :

      If unexpectedly my maxscale crashes it removes existing binlogs from S3 bucket.

        Attachments

          Activity

            People

            Assignee:
            toddstoffel Todd Stoffel
            Reporter:
            ponsuresh.pandians Pon Suresh Pandian
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.