Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-4404

Maxscale: KafkaCDC writes to current_gtid.txt causes high disk utilisation.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 22.08.0
    • 2.5.24, 6.4.4, 22.08.3
    • kafkacdc

    Description

      Hi, the KafkaCDC truncates, then writes, to the current_gtid.txt file for each GTID it processes. The file lives in the Maxscale data directory. We've observed this is causing very high disk utilisation (almost 100%), and double the normal system IOWait. Disk utilisation was literally 0% prior to KafkaCDC. Data appears in Kafka topic to which KafakaCDC writes to, but KafkaCDC cannot keep up with database binary logs as they are purged before reading all of them. The Kafka topic has only one partition. Kafka broker is hosted on a three-host cluster. Database has only three tables, two of which KafkaCDC excludes. Note, to minimise database contention, the transaction binary logs, Galera cache file and database logfile reside on a different virtual disk to what the database resides on.

      Can you provide an option to write the GTID value to memory, instead of/as well as to file?

      1. tail -f /data10/maxscale/Kafka-CDC/current_gtid.txt
        1-1-6459191831
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191834
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191835
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191836
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191839
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191840
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191841
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191842
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191843
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191844
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191845
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191846
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191847
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
        1-1-6459191848
        tail: /data10/maxscale/Kafka-CDC/current_gtid.txt: file truncated
      • Database & Load balancer configs attached.
      • Netdata disk utilisation graph attached.

      Thanks.

      Attachments

        1. my.cnf
          3 kB
        2. maxscale.cnf
          2 kB
        3. KafkaCDC-01.PNG
          KafkaCDC-01.PNG
          121 kB

        Activity

          People

            markus makela markus makela
            Presnickety Presnickety
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.