Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41531

Exponential optimization time for conditions on columns of nested merged derived tables/views

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11
    • 10.11
    • Optimizer
    • None

    Description

      A reference to a column of a merged derived table or view shares the item of the column's expression. With nested derived tables/views referring to columns several times, the item tree becomes a DAG with exponentially many paths.

      MDEV-31130 makes the select list, GROUP BY, HAVING, DISTINCT, ORDER BY, window functions, INSERT and UPDATE traverse such expressions once. When the column is used in WHERE, ON, IN/EXISTS subqueries or a multi-table DELETE condition, the query is still exponential in the nesting depth, and with 20 levels it never finishes.

      Test case (mysqltest)

      create table t1 (f1 int, f2 int, f3 int, f4 int, f5 int);
      insert into t1 values (1,2,3,4,5),(5,4,3,2,1),(3,3,3,3,3);
       
      --disable_query_log
      let $levels= 20;
      let $q= select f1 as a, f1 or f2 as s1 from t1;
      let $i= 1;
      while ($i < $levels)
      {
        let $j= $i;
        inc $j;
        let $q= select a, s$i or s$i or s$i as s$j from ($q) dt$i;
        inc $i;
      }
      let $s= s$levels;
       
      --echo # fast
      eval select count(*) from ($q) x;
      --echo # each of these never finishes
      eval select count(*) from ($q) x where $s > 0;
      eval select count(*) from ($q) x join t1 on x.$s = t1.f3;
      eval select count(*) from t1 left join ($q) x on x.$s = t1.f3;
      eval select count(*) from t1 where f1 in (select $s from ($q) x);
      eval select count(*) from t1 where exists (select 1 from ($q) x where x.$s = t1.f1);
      eval delete t1 from t1, ($q) x where t1.f1 = x.a and x.$s < -1000000;
      --enable_query_log
       
      drop table t1;

      With $levels= 6 all queries succeed with these results: 3, 3, 0, 3, 1, 1.

      Test case (plain SQL)

      create table t1 (f1 int, f2 int, f3 int, f4 int, f5 int);
      insert into t1 values (1,2,3,4,5),(5,4,3,2,1),(3,3,3,3,3);
      -- never finishes; without "where s20 > 0" it takes 0.1 s
      select count(*) from (select a, s19 or s19 or s19 as s20 from (select a, s18 or s18 or s18 as s19 from (select a, s17 or s17 or s17 as s18 from (select a, s16 or s16 or s16 as s17 from (select a, s15 or s15 or s15 as s16 from (select a, s14 or s14 or s14 as s15 from (select a, s13 or s13 or s13 as s14 from (select a, s12 or s12 or s12 as s13 from (select a, s11 or s11 or s11 as s12 from (select a, s10 or s10 or s10 as s11 from (select a, s9 or s9 or s9 as s10 from (select a, s8 or s8 or s8 as s9 from (select a, s7 or s7 or s7 as s8 from (select a, s6 or s6 or s6 as s7 from (select a, s5 or s5 or s5 as s6 from (select a, s4 or s4 or s4 as s5 from (select a, s3 or s3 or s3 as s4 from (select a, s2 or s2 or s2 as s3 from (select a, s1 or s1 or s1 as s2 from (select f1 as a, f1 or f2 as s1 from t1) dt1) dt2) dt3) dt4) dt5) dt6) dt7) dt8) dt9) dt10) dt11) dt12) dt13) dt14) dt15) dt16) dt17) dt18) dt19) x where s20 > 0;

      The same happens with arithmetic, CASE, NULLIF, LIKE, IS NULL and IN() instead of OR.

      Time per nesting depth

      Debug build, 3 rows, OR with 3 references per level:

      depth WHERE ON LEFT JOIN IN EXISTS DELETE
      9 89 ms 80 ms 397 ms 85 ms 94 ms 262 ms
      11 220 ms 144 ms 160 ms 135 ms 188 ms 1251 ms
      13 651 ms 600 ms 829 ms 636 ms 841 ms 10638 ms
      14 1746 ms          
      15 5208 ms          
      16 18662 ms          

      Where the time goes

      • WHERE / ON / EXISTS: walk(eval_not_null_tables) from st_select_lex::update_used_tables()
      • LEFT JOIN: Item_func::fix_after_pullout() from mysql_derived_merge()
      • IN subquery: Item_func::fix_after_pullout() from convert_subq_to_sj()
      • DELETE: Item_func::transform() from Item::top_level_transform() / JOIN::transform_in_predicates_into_equalities()

      walk() of eval_not_null_tables and fix_after_pullout() can be deduplicated like in MDEV-31130 (Item_traversal), but then the query still hangs in transform(). transform() probably cannot just skip a visited item, because it returns the item that replaces it. It would have to transform each shared item once and reuse the result for the other references.

      Evaluating the merged expression per row is exponential as well. This is inherent to merging and is not in the scope of this bug.

      Attachments

        Issue Links

          Activity

            People

              sanja Oleksandr Byelkin
              sanja Oleksandr Byelkin
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.