Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-6821

CDC stats | Latency, average pipeline frequency, backpressure signals

    XMLWordPrintable

Details

    • Task
    • Status: In Progress (View Workflow)
    • Major
    • Resolution: Unresolved
    • 25.10.4
    • 26.10.0
    • exasol-cdc
    • None

    Description

      As a user, i want to lookup the current latency similar to mariadb replication (Seconds_Behind_Master) to know if my CDC is up to date or delayed by x milliseconds.
      Furthermore what the average time to flush a pipeline/micro batch into exasol.
      As well as any indication if the lag is growing or shrinking via a backpressure kpi to understand how far behind we are, maybe how much space is being used because we're behind etc. anything relevant a user would want to know or needs to monitor if CDC cant keep up.

      This information will be used by enterprise manager and mariadb cloud monitoring.

      • Latency — time from source event commit to downstream apply (end-to-end lag), plus per-stage breakdown if feasible (capture, transform, apply).
      • Average pipeline frequency — rate of events/batches processed over a rolling window, to establish a baseline throughput.
      • rows over the last 60 seconds versus mb or kb of data over the last 60s
      • Backpressure signals — indicator(s) when the pipeline is being throttled or queuing due to downstream overload saturation (e.g., queue depth, buffer utilization, or explicit backpressure flag).
      • max RAM over the last hour?

      Attachments

        Issue Links

          Activity

            People

              nantti Niclas Antti
              allen.herrera Allen Herrera
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 3d
                  3d
                  Remaining:
                  Remaining Estimate - 3d
                  3d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.