Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Fix
    • 10.0(EOL)
    • N/A
    • OTHER
    • None

    Description

      hi guys, mariadb-10.0.12 is selecting the wrong index in some cases:

      the difference from both selects is FORCE INDEX (rendimento)
      this one give better results since "lote_rendimento_data" (column) have less values then (item_id,item_id_red) columns

      i don't know if it's a big problem, but this give some slower results, running optimize don't help, running analyze don't help too, i'm not using histograms

      explain
      SELECT DISTINCT "" AS descricao,'+' AS add_sub,"PRODUTOS" AS tabela_preco,d.plano_conta_numero AS centro_numero,d.descricao AS centro_descricao,a.quant AS quant_real,a.pecas AS pecas_real,a.pliq AS pliq_real,a.pbruto AS pbruto_real,a.vliq AS vliq_real,a.vbruto AS vbruto_real,a.quant,a.pecas,a.pliq AS pliq,a.pbruto,a.vliq,a.vbruto,a.custo,a.lote_rendimento_data,a.lote_rendimento_numero,a.oe_tipo,c.plano_conta_numero AS cfop_numero,c.descricao AS cfop_descricao,b.plano_conta_numero AS item_numero,b.descricao AS item_descricao,b.codigo_busca AS item_codigo_busca ,a.unidade_id,a.lote_tipo,a.lote_spa,a.item_id,a.item_id_red,a.mov_id,a.oe,a.oe_seq,(b.plano_conta_id || ";" || b.plano_conta_numero) AS tmp_group,(b.plano_conta_id || ";" || b.codigo_busca) AS tmp_group_cb,60 AS ordem,b.producao_faixa_pliq_1 AS kg_cadastro_1,b.producao_faixa_pliq_2 AS kg_cadastro_2,'cadastro' AS kg_pc_tipo,0 AS kg_pc_1,0 AS kg_pc_2,0 AS valor_venda_un,'un' AS valor_venda_grandeza,0 AS valor_venda_grandeza_quant,(SELECT valor_custo_grandeza_quant FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS valor_custo_grandeza_quant,(SELECT valor_custo_grandeza FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS valor_custo_grandeza,(SELECT mc_minima FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS mc_minima,a.lote_sif_data,a.lote_sif,a.data_estoque FROM est_mov AS a force index (rendimento), estoque_itens AS b, cfop_itens AS c , organograma AS d WHERE a.oe_tipo="tr" AND d.plano_conta_id=a.estoque_entrada_org AND d.plano_conta_id_red=a.estoque_entrada_org_red AND a.estoque_entrada_un=1001 AND d.plano_conta_numero = "3.05.01.01" AND a.item_id=67 AND b.plano_conta_id=a.item_id AND b.plano_conta_id_red=a.item_id_red AND b.codigo_busca LIKE "06%" AND c.plano_conta_id=a.cfop_id AND c.plano_conta_id_red=a.cfop_id_red AND c.plano_conta_id=16 AND c.plano_conta_numero = "0.995" AND a.lote_rendimento_data>="2014-11-13" AND a.lote_rendimento_data<"2014-11-17" AND a.rendimento='Y' AND a.lote_rendimento_numero>=3 AND a.lote_rendimento_numero<=3;

      this one takes 1,997 seconds

      id select_type table type possible_keys key key_len ref rows Extra
      1 PRIMARY c const PRIMARY,NewIndex PRIMARY 265 const,const 1 Using temporary
      1 PRIMARY a range rendimento rendimento 4   171754 Using where
      1 PRIMARY d eq_ref PRIMARY,NewIndex PRIMARY 265 19_org.a.estoque_entrada_org,const 1 Using index condition; Using where
      1 PRIMARY b eq_ref PRIMARY,id,plano_conta_numero,cod_busca id 16 const,19_org.a.item_id_red 1 Using index condition; Using where
      4 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition
      3 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition
      2 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition

      explain
      SELECT DISTINCT "" AS descricao,'+' AS add_sub,"PRODUTOS" AS tabela_preco,d.plano_conta_numero AS centro_numero,d.descricao AS centro_descricao,a.quant AS quant_real,a.pecas AS pecas_real,a.pliq AS pliq_real,a.pbruto AS pbruto_real,a.vliq AS vliq_real,a.vbruto AS vbruto_real,a.quant,a.pecas,a.pliq AS pliq,a.pbruto,a.vliq,a.vbruto,a.custo,a.lote_rendimento_data,a.lote_rendimento_numero,a.oe_tipo,c.plano_conta_numero AS cfop_numero,c.descricao AS cfop_descricao,b.plano_conta_numero AS item_numero,b.descricao AS item_descricao,b.codigo_busca AS item_codigo_busca ,a.unidade_id,a.lote_tipo,a.lote_spa,a.item_id,a.item_id_red,a.mov_id,a.oe,a.oe_seq,(b.plano_conta_id || ";" || b.plano_conta_numero) AS tmp_group,(b.plano_conta_id || ";" || b.codigo_busca) AS tmp_group_cb,60 AS ordem,b.producao_faixa_pliq_1 AS kg_cadastro_1,b.producao_faixa_pliq_2 AS kg_cadastro_2,'cadastro' AS kg_pc_tipo,0 AS kg_pc_1,0 AS kg_pc_2,0 AS valor_venda_un,'un' AS valor_venda_grandeza,0 AS valor_venda_grandeza_quant,(SELECT valor_custo_grandeza_quant FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS valor_custo_grandeza_quant,(SELECT valor_custo_grandeza FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS valor_custo_grandeza,(SELECT mc_minima FROM unidade_itens_custo WHERE unidade_id=a.unidade_id AND item_id=a.item_id AND item_id_red=a.item_id_red) AS mc_minima,a.lote_sif_data,a.lote_sif,a.data_estoque FROM est_mov AS a, estoque_itens AS b, cfop_itens AS c , organograma AS d WHERE a.oe_tipo="tr" AND d.plano_conta_id=a.estoque_entrada_org AND d.plano_conta_id_red=a.estoque_entrada_org_red AND a.estoque_entrada_un=1001 AND d.plano_conta_numero = "3.05.01.01" AND a.item_id=67 AND b.plano_conta_id=a.item_id AND b.plano_conta_id_red=a.item_id_red AND b.codigo_busca LIKE "06%" AND c.plano_conta_id=a.cfop_id AND c.plano_conta_id_red=a.cfop_id_red AND c.plano_conta_id=16 AND c.plano_conta_numero = "0.995" AND a.lote_rendimento_data>="2014-11-13" AND a.lote_rendimento_data<"2014-11-17" AND a.rendimento='Y' AND a.lote_rendimento_numero>=3 AND a.lote_rendimento_numero<=3

      this one takes 2:38 minutes

      id select_type table type possible_keys key key_len ref rows Extra
      1 PRIMARY c const PRIMARY,NewIndex PRIMARY 265 const,const 1 Using temporary
      1 PRIMARY b range PRIMARY,id,plano_conta_numero,cod_busca cod_busca 265   53 Using index condition
      1 PRIMARY a ref cfop,item,transferencias,rendimento,estoque,giro item 6 const,19_org.b.plano_conta_id_red 3316 Using where
      1 PRIMARY d eq_ref PRIMARY,NewIndex PRIMARY 265 19_org.a.estoque_entrada_org,const 1 Using index condition; Using where
      4 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition
      3 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition
      2 DEPENDENT SUBQUERY unidade_itens_custo eq_ref PRIMARY PRIMARY 24 19_org.a.unidade_id,19_org.a.item_id,19_org.a.item_id_red 1 Using index condition

      Attachments

        Activity

          i didn't tested... item_id/item_id_red columns are part of primary key, if you remove primary key and leave item and rendimento index, optimizer prefer item index?

          rspadim roberto spadim added a comment - i didn't tested... item_id/item_id_red columns are part of primary key, if you remove primary key and leave item and rendimento index, optimizer prefer item index?
          rspadim roberto spadim added a comment - - edited

          from "show index from est_mov_myisam":

          Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment
          est_mov_myisam 1 rendimento 1 rendimento A 3       BTREE    
          est_mov_myisam 1 rendimento 2 lote_rendimento_data A 6078       BTREE    
          est_mov_myisam 1 rendimento 3 item_id_red A 832974       BTREE    
          est_mov_myisam 1 item 1 item_id A 10       BTREE    
          est_mov_myisam 1 item 2 item_id_red A 8786       BTREE    
          est_mov_myisam 1 item 3 data_estoque A 14577050       BTREE    
          rspadim roberto spadim added a comment - - edited from "show index from est_mov_myisam": Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment est_mov_myisam 1 rendimento 1 rendimento A 3       BTREE     est_mov_myisam 1 rendimento 2 lote_rendimento_data A 6078       BTREE     est_mov_myisam 1 rendimento 3 item_id_red A 832974       BTREE     est_mov_myisam 1 item 1 item_id A 10       BTREE     est_mov_myisam 1 item 2 item_id_red A 8786       BTREE     est_mov_myisam 1 item 3 data_estoque A 14577050       BTREE    

          psergey,

          As requested, results (no need to go through previous comments, they are intermediate steps).

          • The problem is that the query uses ref access with the index `item` on the table `a`, while range with index `rendimento` provides significantly better performance for this query. It is reproducible. Depending on the query, performance difference varied from 1.5x to 30x and more (in some cases I didn't wait for the "bad" query to finish).
          • The query is simplified (see below) to leave only one join and only AND conditions. No subqueries. Also, all select items have been replaced by a single COUNT to make the result set nicer.
          • The table structures are simplified to only have columns and indexes that directly participate in the query. Also, tables are switched from Aria to MyISAM, and partitioning is removed.
          • The data for the second table is reduced to 2 rows.
          • The data for the first table (the one with indexes in question) is shortened a bit, to 20 mln rows (the original data was 29 mln rows). It's also possible to reduce it to some 15 mln and still have the difference in plans, but the performance difference becomes negligible

          At this point, the performance difference on my machine is ~3x (0.13 vs 0.35 sec). On another machine ~4x (0.07 vs 0.30 sec). This is for "warm" queries, cold ones can take considerably longer and are less predictable.
          Below the "BAD" query is the one where the wrong index/access is used. The "GOOD" query is where the desired index is enforced.

          MariaDB [test]> select count(*) from t1_min;
          +----------+
          | count(*) |
          +----------+
          | 20000000 |
          +----------+
          1 row in set (0.00 sec)
           
          MariaDB [test]> select count(*) from t2_min;
          +----------+
          | count(*) |
          +----------+
          |        2 |
          +----------+
          1 row in set (0.00 sec)

          CREATE TABLE `t1_min` (
            `item_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
            `item_id_red` mediumint(9) NOT NULL DEFAULT '0',
            `lote_rendimento_data` date NOT NULL DEFAULT '0000-00-00',
            `rendimento` enum('Y','N') NOT NULL DEFAULT 'N',
            KEY `item` (`item_id`,`item_id_red`),
            KEY `rendimento` (`rendimento`,`lote_rendimento_data`,`item_id_red`)
          ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
           
          CREATE TABLE `t2_min` (
            `plano_conta_id_red` bigint(20) NOT NULL DEFAULT '0'
          ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

          MariaDB [test]> show index in t1_min;
          +--------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
          | Table  | Non_unique | Key_name   | Seq_in_index | Column_name          | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
          +--------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
          | t1_min |          1 | item       |            1 | item_id              | A         |           5 |     NULL | NULL   |      | BTREE      |         |               |
          | t1_min |          1 | item       |            2 | item_id_red          | A         |        4672 |     NULL | NULL   |      | BTREE      |         |               |
          | t1_min |          1 | rendimento |            1 | rendimento           | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
          | t1_min |          1 | rendimento |            2 | lote_rendimento_data | A         |        3518 |     NULL | NULL   |      | BTREE      |         |               |
          | t1_min |          1 | rendimento |            3 | item_id_red          | A         |      338983 |     NULL | NULL   |      | BTREE      |         |               |
          +--------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

          SELECT COUNT(*) AS GOOD FROM t1_min a force index(rendimento), t2_min AS b WHERE a.item_id=67 AND b.plano_conta_id_red=a.item_id_red AND a.lote_rendimento_data>="2014-10-13" AND a.lote_rendimento_data<"2014-10-17" AND a.rendimento='Y';
          +------+
          | GOOD |
          +------+
          |  863 |
          +------+
          1 row in set (0.13 sec)
           
          | id   | select_type | table | type  | possible_keys | key        | key_len | ref  | rows  | filtered | Extra                                                                  |
          +------+-------------+-------+-------+---------------+------------+---------+------+-------+----------+------------------------------------------------------------------------+
          |    1 | SIMPLE      | b     | ALL   | NULL          | NULL       | NULL    | NULL |     2 |   100.00 |                                                                        |
          |    1 | SIMPLE      | a     | range | rendimento    | rendimento | 4       | NULL | 52529 |    75.00 | Using index condition; Using where; Using join buffer (flat, BNL join) |
           
          | Note  | 1003 | select count(0) AS `GOOD` from `test`.`t1_min` `a` FORCE INDEX (`rendimento`) join `test`.`t2_min` `b` where ((`test`.`a`.`item_id` = 67) and (`test`.`a`.`rendimento` = 'Y') and (`test`.`b`.`plano_conta_id_red` = `test`.`a`.`item_id_red`) and (`test`.`a`.`lote_rendimento_data` >= '2014-10-13') and (`test`.`a`.`lote_rendimento_data` < '2014-10-17')) |

          SELECT COUNT(*) AS BAD FROM t1_min a, t2_min AS b WHERE a.item_id=67 AND b.plano_conta_id_red=a.item_id_red AND a.lote_rendimento_data>="2014-10-13" AND a.lote_rendimento_data<"2014-10-17" AND a.rendimento='Y';
          +-----+
          | BAD |
          +-----+
          | 863 |
          +-----+
          1 row in set (0.35 sec)
           
          | id   | select_type | table | type | possible_keys   | key  | key_len | ref                             | rows | filtered | Extra                              |
          +------+-------------+-------+------+-----------------+------+---------+---------------------------------+------+----------+------------------------------------+
          |    1 | SIMPLE      | b     | ALL  | NULL            | NULL | NULL    | NULL                            |    2 |   100.00 |                                    |
          |    1 | SIMPLE      | a     | ref  | item,rendimento | item | 6       | const,test.b.plano_conta_id_red | 4280 |   100.00 | Using index condition; Using where |
           
          | Note  | 1003 | select count(0) AS `BAD` from `test`.`t1_min` `a` join `test`.`t2_min` `b` where ((`test`.`a`.`item_id` = 67) and (`test`.`a`.`rendimento` = 'Y') and (`test`.`b`.`plano_conta_id_red` = `test`.`a`.`item_id_red`) and (`test`.`a`.`lote_rendimento_data` >= '2014-10-13') and (`test`.`a`.`lote_rendimento_data` < '2014-10-17')) |

          I've uploaded the new dump to ftp.askmonty.org/private/mdev7125-smaller.dump.gz.

          elenst Elena Stepanova added a comment - psergey , As requested, results (no need to go through previous comments, they are intermediate steps). The problem is that the query uses ref access with the index `item` on the table `a`, while range with index `rendimento` provides significantly better performance for this query. It is reproducible. Depending on the query, performance difference varied from 1.5x to 30x and more (in some cases I didn't wait for the "bad" query to finish). The query is simplified (see below) to leave only one join and only AND conditions. No subqueries. Also, all select items have been replaced by a single COUNT to make the result set nicer. The table structures are simplified to only have columns and indexes that directly participate in the query. Also, tables are switched from Aria to MyISAM, and partitioning is removed. The data for the second table is reduced to 2 rows. The data for the first table (the one with indexes in question) is shortened a bit, to 20 mln rows (the original data was 29 mln rows). It's also possible to reduce it to some 15 mln and still have the difference in plans, but the performance difference becomes negligible At this point, the performance difference on my machine is ~3x (0.13 vs 0.35 sec). On another machine ~4x (0.07 vs 0.30 sec). This is for "warm" queries, cold ones can take considerably longer and are less predictable. Below the "BAD" query is the one where the wrong index/access is used. The "GOOD" query is where the desired index is enforced. MariaDB [test]> select count (*) from t1_min; + ----------+ | count (*) | + ----------+ | 20000000 | + ----------+ 1 row in set (0.00 sec)   MariaDB [test]> select count (*) from t2_min; + ----------+ | count (*) | + ----------+ | 2 | + ----------+ 1 row in set (0.00 sec) CREATE TABLE `t1_min` ( `item_id` mediumint(8) unsigned NOT NULL DEFAULT '0' , `item_id_red` mediumint(9) NOT NULL DEFAULT '0' , `lote_rendimento_data` date NOT NULL DEFAULT '0000-00-00' , `rendimento` enum( 'Y' , 'N' ) NOT NULL DEFAULT 'N' , KEY `item` (`item_id`,`item_id_red`), KEY `rendimento` (`rendimento`,`lote_rendimento_data`,`item_id_red`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1;   CREATE TABLE `t2_min` ( `plano_conta_id_red` bigint (20) NOT NULL DEFAULT '0' ) ENGINE=MyISAM DEFAULT CHARSET=latin1; MariaDB [test]> show index in t1_min; + --------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | + --------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | t1_min | 1 | item | 1 | item_id | A | 5 | NULL | NULL | | BTREE | | | | t1_min | 1 | item | 2 | item_id_red | A | 4672 | NULL | NULL | | BTREE | | | | t1_min | 1 | rendimento | 1 | rendimento | A | 2 | NULL | NULL | | BTREE | | | | t1_min | 1 | rendimento | 2 | lote_rendimento_data | A | 3518 | NULL | NULL | | BTREE | | | | t1_min | 1 | rendimento | 3 | item_id_red | A | 338983 | NULL | NULL | | BTREE | | | + --------+------------+------------+--------------+----------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ SELECT COUNT (*) AS GOOD FROM t1_min a force index (rendimento), t2_min AS b WHERE a.item_id=67 AND b.plano_conta_id_red=a.item_id_red AND a.lote_rendimento_data>= "2014-10-13" AND a.lote_rendimento_data< "2014-10-17" AND a.rendimento= 'Y' ; + ------+ | GOOD | + ------+ | 863 | + ------+ 1 row in set (0.13 sec)   | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | + ------+-------------+-------+-------+---------------+------------+---------+------+-------+----------+------------------------------------------------------------------------+ | 1 | SIMPLE | b | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | | | 1 | SIMPLE | a | range | rendimento | rendimento | 4 | NULL | 52529 | 75.00 | Using index condition; Using where ; Using join buffer (flat, BNL join ) |   | Note | 1003 | select count (0) AS `GOOD` from `test`.`t1_min` `a` FORCE INDEX (`rendimento`) join `test`.`t2_min` `b` where ((`test`.`a`.`item_id` = 67) and (`test`.`a`.`rendimento` = 'Y' ) and (`test`.`b`.`plano_conta_id_red` = `test`.`a`.`item_id_red`) and (`test`.`a`.`lote_rendimento_data` >= '2014-10-13' ) and (`test`.`a`.`lote_rendimento_data` < '2014-10-17' )) | SELECT COUNT (*) AS BAD FROM t1_min a, t2_min AS b WHERE a.item_id=67 AND b.plano_conta_id_red=a.item_id_red AND a.lote_rendimento_data>= "2014-10-13" AND a.lote_rendimento_data< "2014-10-17" AND a.rendimento= 'Y' ; + -----+ | BAD | + -----+ | 863 | + -----+ 1 row in set (0.35 sec)   | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | + ------+-------------+-------+------+-----------------+------+---------+---------------------------------+------+----------+------------------------------------+ | 1 | SIMPLE | b | ALL | NULL | NULL | NULL | NULL | 2 | 100.00 | | | 1 | SIMPLE | a | ref | item,rendimento | item | 6 | const,test.b.plano_conta_id_red | 4280 | 100.00 | Using index condition; Using where |   | Note | 1003 | select count (0) AS `BAD` from `test`.`t1_min` `a` join `test`.`t2_min` `b` where ((`test`.`a`.`item_id` = 67) and (`test`.`a`.`rendimento` = 'Y' ) and (`test`.`b`.`plano_conta_id_red` = `test`.`a`.`item_id_red`) and (`test`.`a`.`lote_rendimento_data` >= '2014-10-13' ) and (`test`.`a`.`lote_rendimento_data` < '2014-10-17' )) | I've uploaded the new dump to ftp.askmonty.org/private/mdev7125-smaller.dump.gz.

          hi guys any news?

          rspadim roberto spadim added a comment - hi guys any news?

          10.0 was EOLed in March 2019

          serg Sergei Golubchik added a comment - 10.0 was EOLed in March 2019

          People

            psergei Sergei Petrunia
            rspadim roberto spadim
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.