ColumnStore Engine Monitoring and Metrics

If/when we have a working production cluster, we will need to monitor it. We use Datadog for observability with all of our deployments, which includes an agent integration that generates health and performance metrics by periodically scraping the system tables (e.g., performance_schema, sys) of MySQL-compatible databases.

This is the list of metrics we are collecting from our MySQL databases: https://docs.datadoghq.com/integrations/mysql/?tab=host#data-collected

Naturally, this integration is unaware of the ColumnStore engine because it is meant for MySQL, but it can be extended with arbitrary queries to scrape additional information from the target database, which can then also be recorded as health checks and timeseries metrics.

We would like to ask whether there exist similar tables that express meaningful information specific to the internal state of the ColumnStore engine that we can use to measure its key performance indicators, and if so, what you would consider to be the most vital metrics to collect.

I've searched for just about anything we could query from the Columnstore engine that is comparable in quality to the summarize data one can easily query from the InnoDB and MyISAM engines. The only thing I found was this:

https://mariadb.com/kb/en/analyzing-queries-in-columnstore/#query-statistics-table

Some of those data points would be very useful to us, but having followed the instructions to enable it, I find that the "QueryStats" parameter in Columnstore.xml has no effect at all. There is even a comment at the bottom of that KB article more than four years old indicating someone else had the same experience on a much older version, so it isn't clear at all whether that feature is in fact supposed to work or whether it is just another holdover from InfiniDB (there seem to be quite a lot of those).

How then do you propose we query the values of Columnstore-specific resource utilizations and statistics, such as:

* outstanding requests

* engine memory utilization

* per-table and per-schema storage utilization

* engine storage layer throughput (data read, written, cached, pending, etc.)

* engine row/column operations (discrete reads, writes, locks)

* disk-based table join temporary storage utilization

* valid/invalid extents

etc.