Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4821

ColumnStore Engine Monitoring and Metrics

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.6.1
    • Fix Version/s: Icebox
    • Component/s: Documentation
    • Labels:
      None

      Description

      Here is the customer's questions on CS monitoring and metrics. I think it is more like doc's team work.

      If/when we have a working production cluster, we will need to monitor it. We use Datadog for observability with all of our deployments, which includes an agent integration that generates health and performance metrics by periodically scraping the system tables (e.g., performance_schema, sys) of MySQL-compatible databases.
       
      This is the list of metrics we are collecting from our MySQL databases: https://docs.datadoghq.com/integrations/mysql/?tab=host#data-collected
       
      Naturally, this integration is unaware of the ColumnStore engine because it is meant for MySQL, but it can be extended with arbitrary queries to scrape additional information from the target database, which can then also be recorded as health checks and timeseries metrics.
       
      We would like to ask whether there exist similar tables that express meaningful information specific to the internal state of the ColumnStore engine that we can use to measure its key performance indicators, and if so, what you would consider to be the most vital metrics to collect.
       
      I've searched for just about anything we could query from the Columnstore engine that is comparable in quality to the summarize data one can easily query from the InnoDB and MyISAM engines. The only thing I found was this:
       
      https://mariadb.com/kb/en/analyzing-queries-in-columnstore/#query-statistics-table
       
      Some of those data points would be very useful to us, but having followed the instructions to enable it, I find that the "QueryStats" parameter in Columnstore.xml has no effect at all. There is even a comment at the bottom of that KB article more than four years old indicating someone else had the same experience on a much older version, so it isn't clear at all whether that feature is in fact supposed to work or whether it is just another holdover from InfiniDB (there seem to be quite a lot of those).
       
      How then do you propose we query the values of Columnstore-specific resource utilizations and statistics, such as:
       
      * outstanding requests
      * engine memory utilization
      * per-table and per-schema storage utilization
      * engine storage layer throughput (data read, written, cached, pending, etc.)
      * engine row/column operations (discrete reads, writes, locks)
      * disk-based table join temporary storage utilization
      * valid/invalid extents
      etc.
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              toddstoffel Todd Stoffel
              Reporter:
              allen.lee@mariadb.com Allen Lee
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.