[MDEV-27021] Extend SHOW EXPLAIN to support SHOW ANALYZE [FORMAT=JSON] Created: 2021-11-11  Updated: 2024-01-22  Resolved: 2022-04-29

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Fix Version/s: 10.9.1

Type: Task Priority: Major
Reporter: Sergei Petrunia Assignee: Alice Sherepa
Resolution: Fixed Votes: 0
Labels: Preview_10.9

Issue Links:
PartOf
is part of MDEV-28112 prepare 10.9.0 preview releases Closed
Problem/Incident
causes MDEV-28124 Server crashes in Explain_aggr_fileso... Closed
causes MDEV-28125 Server crashes in Explain_table_acces... Closed
causes MDEV-28201 Server crashes upon SHOW ANALYZE/EXPL... Closed
causes MDEV-30156 SHOW ANALYZE is missing a KB page Closed
causes MDEV-31774 mariadb command line client missing s... Open
causes MDEV-33293 Race condition in test main.show_analyze Open
Relates
relates to MDEV-10000 Add EXPLAIN FOR CONNECTION syntax sup... Closed
relates to MDEV-25956 SHOW EXPLAIN FOR should support FORMA... Closed
relates to MDEV-28318 SHOW ANALYZE|EXPLAIN: Lock wait timeo... Stalled

 Description   

SHOW EXPLAIN allows one to view EXPLAIN of a currently executing query.

However, it doesn't handle this use case:

A really huge query runs for a long time and never finishes. Since it never finishes, one cannot view ANALYZE output. Adding "LIMIT N" to the top query is of limited help.
One can view the EXPLAIN, but the query has a lot of subqueries that have many-table joins. One can see multiple sub-optimal access methods, and it is not clear which of them are the ones that are causing the performance problem.

Suggested solution: extend SHOW EXPLAIN to report SHOW ANALYZE FORMAT=JSON data of the query that's currently running.



 Comments   
Comment by Sergei Petrunia [ 2022-01-28 ]

Review input: https://lists.launchpad.net/maria-developers/msg13078.html .

The tabular form is basically done, need to do the FORMAT=JSON part.

Comment by Sergei Petrunia [ 2022-02-08 ]

Trying the FORMAT=JSON on a test query: https://gist.github.com/spetrunia/69fb0c48b3b5412a546575f02d3b3de7

Ok this gives some clue about query execution. Things that I would like to see but do not see

  • a clear indication that this is a SHOW ANALYZE output, that is, it describes a run in progress. (reasoning: we expect to get it from users/customers/etc. need to minimize the effects of getting the wrong file and other kinds of "broken telephone" effect).
  • how much time the query has spent running (this is needed to relate it with other r_*time_ms values).
Comment by Sergei Petrunia [ 2022-02-08 ]

A select is in its first execution looks like so:

          "query_block": {
            "select_id": 2,
            "operation": "UNION",
            "r_loops": 1,
            "r_total_time_ms": 0,

compare to select that has never been executed:

          "query_block": {
            "select_id": 3,
            "nested_loop": [

...

Comment by Oleg Smirnov [ 2022-02-09 ]

1. How would you like to see the clear indication?

  • SHOW ANALYZE: {
  • SHOW ANALYZE (running query): {
  • any other?

2. How to name this field and where to place it? Is it OK to place it in the top of the output like this:
SHOW ANALYZE: {
"r_query_running_time_ms": 5432
"query_block": {
"select_id": 1,
...

Comment by Sergei Petrunia [ 2022-02-11 ]

oleg.smirnov, good questions.. I'm still trying to figure out answer for these...

Comment by Sergei Petrunia [ 2022-02-11 ]

Looking at a query that does filesort first, and then join with the second table:
https://gist.github.com/spetrunia/cb8c73f62d733a771697aade575a54c0

(diff'ing first SHOW ANALYZE output against the second)

Comment by Sergei Petrunia [ 2022-02-11 ]

Here's an example : https://gist.github.com/spetrunia/664a80db9110b02e0a91642e8aa987df
The target runs a regular SELECT - the execution is not timed.
However, the output has r_total_time_ms members:

          "query_block": {
            "select_id": 2,
            "r_loops": 731,
            "r_total_time_ms": 0,

I'm not sure if we should call this incorrect, but looking at it as a user, it doesn't look very intuitive...

Comment by Sergei Petrunia [ 2022-02-14 ]

Indeed, r_total_time_ms shows up due to this logic in Explain_select::print_explain_json:

    if (is_analyze && time_tracker.get_loops())
    {
      writer->add_member("r_loops").add_ll(time_tracker.get_loops());
      writer->add_member("r_total_time_ms").add_double(time_tracker.get_time_ms());
    }

The code assumes that is_analyze=true means that there is timing information.

The first idea off the top of the head: check if time_tracker.cycles==0 (wrap this into a call like time_tracker.have_timings()) and don't print r_*time_ms in this case.

Comment by Sergei Petrunia [ 2022-02-14 ]

2. How to name this field and where to place it? Is it OK to place it in the top of the output like this:
SHOW ANALYZE: {
"r_query_running_time_ms": 5432
"query_block": {

As agreed on Slack: Let it be r_time_in_progress_ms at the top level. For non-ANALYZE execution, there's a question of where to get the query start time. Please check where INFORMATION_SCHEMA.PROCESSLIST gets it from.

Comment by Oleg Smirnov [ 2022-02-16 ]

Pushed a new commit addressing the issues discussed.

{{commit a335245bbf9c49327d8b34ac30e3ad49822a5aa3 (HEAD > bb-10.8MDEV-10000, origin/bb-10.8-MDEV-10000)
Author: Oleg Smirnov <olernov@gmail.com>
Date: Wed Feb 16 13:03:46 2022 +0700

MDEV-27021 Add explicit indication of SHOW EXPLAIN/ANALYZE.

1. Add explicit indication that the output is produced by
SHOW EXPLAIN/ANALYZE FORMAT=JSON command.
2. Remove useless "r_total_time_ms" field from SHOW ANALYZE FORMAT=JSON
output when there is no timed statistics gathered.
3. Add "r_query_time_in_progress_ms" to the output of SHOW ANALYZE FORMAT=JSON.
}}

psergei, please review.

Comment by Oleg Smirnov [ 2022-02-16 ]

By the way, I haven't fixed test cases yet, so some tests will fail. Once we're happy with the results I'll fix the test cases accordingly.

Comment by Sergei Petrunia [ 2022-03-01 ]

The last patch is ok. I've requested small fix in comments. After that, we can pass it to testing.

Comment by Nuno [ 2022-04-16 ]

Hey
Nice to see this being implemented!

Here's a related post:

"extend explain output to include "attached_condition""
https://jira.mariadb.org/browse/MDEV-27146

Thanks!

Generated at Thu Feb 08 09:49:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.