[MDEV-16695] Estimate for rows of derived tables is very high when we are using index_merge union Created: 2018-07-05  Updated: 2018-11-05  Resolved: 2018-11-01

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.0, 10.1, 10.2, 10.3, 10.4
Fix Version/s: 10.3.11, 10.2.19, 10.1.38, 10.0.38

Type: Bug Priority: Major
Reporter: Varun Gupta (Inactive) Assignee: Varun Gupta (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
is part of MDEV-15253 Default optimizer setting changes for... Closed

 Description   

Here is the dataset

create table t0
(
  key1 int not null,
  INDEX i1(key1)
);
 
insert into t0 values (1),(2),(3),(4),(5),(6),(7),(8);
 
let $1=7;
set @d=8;
while ($1)
{
  eval insert into t0 select key1+@d from t0;
  eval set @d=@d*2;
  dec $1;
}
 
alter table t0 add key2 int not null, add index i2(key2);

analyze format=json select * from (select * from t0 where key1 = 3 or key2 =3) as Z ;
ANALYZE
{
  "query_block": {
    "select_id": 1,
    "r_loops": 1,
    "r_total_time_ms": 0.5064,
    "table": {
      "table_name": "<derived2>",
      "access_type": "ALL",
      "r_loops": 1,
      "rows": 1024,
      "r_rows": 1,
      "r_total_time_ms": 0.0103,
      "filtered": 100,
      "r_filtered": 100,
      "materialized": {
        "query_block": {
          "select_id": 2,
          "r_loops": 1,
          "r_total_time_ms": 0.4476,
          "table": {
            "table_name": "t0",
            "access_type": "index_merge",
            "possible_keys": ["i1", "i2"],
            "key_length": "4,4",
            "index_merge": {
              "union": {
                "range": {
                  "key": "i1",
                  "used_key_parts": ["key1"]
                },
                "range": {
                  "key": "i2",
                  "used_key_parts": ["key2"]
                }
              }
            },
            "r_loops": 1,
            "rows": 2,
            "r_rows": 1,
            "r_total_time_ms": 0.0096,
            "filtered": 100,
            "r_filtered": 100,
            "attached_condition": "t0.key1 = 3 or t0.key2 = 3"
          }
        }
      }
    }
  }
}

alter table t0 add key3 int not null, add index i3(key3);
alter table t0 add key8 int not null, add index i8(key8);
update t0 set key2=key1,key3=key1,key8=1024-key1;
analyze table t0;
set optimizer_switch='derived_merge=off,derived_with_keys=off';

 

set optimizer_use_condition_selectivity=2;

analyze format=json select * from (select * from t0 where key1 = 3 or key2 =3) as Z ;
ANALYZE
{
  "query_block": {
    "select_id": 1,
    "r_loops": 1,
    "r_total_time_ms": 0.5064,
    "table": {
      "table_name": "<derived2>",
      "access_type": "ALL",
      "r_loops": 1,
      "rows": 1024,
      "r_rows": 1,
      "r_total_time_ms": 0.0103,
      "filtered": 100,
      "r_filtered": 100,
      "materialized": {
        "query_block": {
          "select_id": 2,
          "r_loops": 1,
          "r_total_time_ms": 0.4476,
          "table": {
            "table_name": "t0",
            "access_type": "index_merge",
            "possible_keys": ["i1", "i2"],
            "key_length": "4,4",
            "index_merge": {
              "union": {
                "range": {
                  "key": "i1",
                  "used_key_parts": ["key1"]
                },
                "range": {
                  "key": "i2",
                  "used_key_parts": ["key2"]
                }
              }
            },
            "r_loops": 1,
            "rows": 2,
            "r_rows": 1,
            "r_total_time_ms": 0.0096,
            "filtered": 100,
            "r_filtered": 100,
            "attached_condition": "t0.key1 = 3 or t0.key2 = 3"
          }
        }
      }
    }
  }
}

set optimizer_use_condition_selectivity=1;

analyze format=json select * from (select * from t0 where key1 = 3 or key2 =3) as Z ;
ANALYZE
{
  "query_block": {
    "select_id": 1,
    "r_loops": 1,
    "r_total_time_ms": 0.3456,
    "table": {
      "table_name": "<derived2>",
      "access_type": "ALL",
      "r_loops": 1,
      "rows": 2,
      "r_rows": 1,
      "r_total_time_ms": 0.0092,
      "filtered": 100,
      "r_filtered": 100,
      "materialized": {
        "query_block": {
          "select_id": 2,
          "r_loops": 1,
          "r_total_time_ms": 0.3075,
          "table": {
            "table_name": "t0",
            "access_type": "index_merge",
            "possible_keys": ["i1", "i2"],
            "key_length": "4,4",
            "index_merge": {
              "union": {
                "range": {
                  "key": "i1",
                  "used_key_parts": ["key1"]
                },
                "range": {
                  "key": "i2",
                  "used_key_parts": ["key2"]
                }
              }
            },
            "r_loops": 1,
            "rows": 2,
            "r_rows": 1,
            "r_total_time_ms": 0.0089,
            "filtered": 100,
            "r_filtered": 100,
            "attached_condition": "t0.key1 = 3 or t0.key2 = 3"
          }
        }
      }
    }
  }
}

The issue we see is when we have optimizer_use_condition_selectivity set to 2 the derived table has rows =1024 but index merge predicts that there would be 2 rows but looks that during calculation of selectivity we don't take into consideration the rows predicited by index merge



 Comments   
Comment by Varun Gupta (Inactive) [ 2018-07-05 ]

For index_merge/intersection
it is made of multiple range accesses and each of them already has contributed to the estimate of table->cond_selectivity.

table->cond_selectivity= table->cond_selectivity *( ( quick_cond_selectivity(index1) *quick_cond_selectivity(index2)* ... ))

For index_merge/union
the selectivity of index_merge union is not counted in the selectivity estimate

Comment by Varun Gupta (Inactive) [ 2018-07-09 ]

The issue has been that the calculate of selectivity for the condition does not include the selectivity of index merge[union or sort union].

Comment by Varun Gupta (Inactive) [ 2018-07-09 ]

http://lists.askmonty.org/pipermail/commits/2018-July/012688.html

Comment by Sergei Petrunia [ 2018-08-26 ]

Ok to push

Comment by Sergei Petrunia [ 2018-08-26 ]

Wait this gets confused with another issue.

Comment by Sergei Petrunia [ 2018-10-25 ]

Ok to push this one.

Generated at Thu Feb 08 08:30:52 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.