Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32575

MSAN / Valgrind errors in test_if_cheaper_ordering upon reaching in_predicate_conversion_threshold

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 11.0(EOL), 11.1(EOL), 11.2(EOL)
    • 11.4.5, 11.7.2
    • Optimizer
    • None

    Description

      CREATE TABLE lineitem (l_orderkey INT, l_linenumber INT, l_quantity INT, PRIMARY KEY (l_orderkey,l_linenumber));
      		 
       
      		 
      INSERT INTO lineitem (l_orderkey,l_linenumber) VALUES 
        (261,3),(261,4),(261,5),(261,6),(262,1),(262,2),(262,3),(263,1);
      		 
       
      		 
      SET in_predicate_conversion_threshold= 2;
      		 
      SELECT * FROM lineitem WHERE l_orderkey IN (248, 250) AND l_linenumber = 3 ORDER BY l_orderkey, l_linenumber;
      		 
       
      		 
      # Cleanup
      		 
      DROP TABLE lineitem;

      11.0 be24e75229a4496e525677149c8a116a3ff4b372 MSAN

      		 
      ==503076==WARNING: MemorySanitizer: use-of-uninitialized-value
      		 
          #0 0x56131e53ce01 in test_if_cheaper_ordering(st_join_table const*, st_order*, TABLE*, Bitmap<64u>, int, unsigned long long, int*, int*, unsigned long long*, unsigned int*, unsigned int*) /data/src/11.0-msan/sql/sql_select.cc:31820:25
      		 
          #1 0x56131e434cbf in test_if_skip_sort_order(st_join_table*, st_order*, unsigned long long, bool, Bitmap<64u> const*) /data/src/11.0-msan/sql/sql_select.cc:26463:5
      		 
          #2 0x56131e3f243d in JOIN::optimize_stage2() /data/src/11.0-msan/sql/sql_select.cc:3348:11
      		 
          #3 0x56131e3fdc76 in JOIN::optimize_inner() /data/src/11.0-msan/sql/sql_select.cc:2635:9
      		 
          #4 0x56131e3e0bf0 in JOIN::optimize() /data/src/11.0-msan/sql/sql_select.cc:1935:10
      		 
          #5 0x56131e3c0955 in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /data/src/11.0-msan/sql/sql_select.cc:5176:19
      		 
          #6 0x56131e3c0076 in handle_select(THD*, LEX*, select_result*, unsigned long long) /data/src/11.0-msan/sql/sql_select.cc:626:10
      		 
          #7 0x56131e29390d in execute_sqlcom_select(THD*, TABLE_LIST*) /data/src/11.0-msan/sql/sql_parse.cc:6290:12
      		 
          #8 0x56131e26d12d in mysql_execute_command(THD*, bool) /data/src/11.0-msan/sql/sql_parse.cc:3961:12
      		 
          #9 0x56131e254a5a in mysql_parse(THD*, char*, unsigned int, Parser_state*) /data/src/11.0-msan/sql/sql_parse.cc:8027:18
      		 
          #10 0x56131e248f50 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /data/src/11.0-msan/sql/sql_parse.cc:1894:7
      		 
          #11 0x56131e256e50 in do_command(THD*, bool) /data/src/11.0-msan/sql/sql_parse.cc:1407:17
      		 
          #12 0x56131e91a3af in do_handle_one_connection(CONNECT*, bool) /data/src/11.0-msan/sql/sql_connect.cc:1416:11
      		 
          #13 0x56131e919985 in handle_one_connection /data/src/11.0-msan/sql/sql_connect.cc:1318:5
      		 
          #14 0x56131fcb78aa in pfs_spawn_thread /data/src/11.0-msan/storage/perfschema/pfs.cc:2201:3
      		 
          #15 0x7fb1bdfdcfd3 in start_thread nptl/./nptl/pthread_create.c:442:8
      		 
          #16 0x7fb1be05d5bb in clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
      		 
       
      		 
        Memory was marked as uninitialized
      		 
          #0 0x56131dc2021e in __msan_allocated_memory (/mnt8t/src/11.0-msan/sql/mariadbd+0x7a721e)
      		 
          #1 0x561320f9e1b2 in my_malloc /data/src/11.0-msan/mysys/my_malloc.c:119:7
      		 
       
      		 
      SUMMARY: MemorySanitizer: use-of-uninitialized-value /data/src/11.0-msan/sql/sql_select.cc:31820:25 in test_if_cheaper_ordering(st_join_table const*, st_order*, TABLE*, Bitmap<64u>, int, unsigned long long, int*, int*, unsigned long long*, unsigned int*, unsigned int*)

      11.0 5e2d08b5 Valgrind

      		 
      ==504192== Thread 6:
      		 
      ==504192== Conditional jump or move depends on uninitialised value(s)
      		 
      ==504192==    at 0xBE29AA: test_if_cheaper_ordering(st_join_table const*, st_order*, TABLE*, Bitmap<64u>, int, unsigned long long, int*, int*, unsigned long long*, unsigned int*, unsigned int*) (sql_select.cc:31795)
      		 
      ==504192==    by 0xBD2D05: test_if_skip_sort_order(st_join_table*, st_order*, unsigned long long, bool, Bitmap<64u> const*) (sql_select.cc:26438)
      		 
      ==504192==    by 0xB8F92F: JOIN::optimize_stage2() (sql_select.cc:3348)
      		 
      ==504192==    by 0xB8D045: JOIN::optimize_inner() (sql_select.cc:2635)
      		 
      ==504192==    by 0xB8A662: JOIN::optimize() (sql_select.cc:1935)
      		 
      ==504192==    by 0xB967AD: mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:5176)
      		 
      ==504192==    by 0xB84DE7: handle_select(THD*, LEX*, select_result*, unsigned long long) (sql_select.cc:626)
      		 
      ==504192==    by 0xB2842C: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:6288)
      		 
      ==504192==    by 0xB1F2C1: mysql_execute_command(THD*, bool) (sql_parse.cc:3959)
      		 
      ==504192==    by 0xB2D311: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:8030)
      		 
      ==504192==    by 0xB18D9E: dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) (sql_parse.cc:1894)
      		 
      ==504192==    by 0xB176F7: do_command(THD*, bool) (sql_parse.cc:1407)
      		 
      ==504192==    by 0xD1448F: do_handle_one_connection(CONNECT*, bool) (sql_connect.cc:1416)
      		 
      ==504192==    by 0xD14204: handle_one_connection (sql_connect.cc:1318)
      		 
      ==504192==    by 0x124ADED: pfs_spawn_thread (pfs.cc:2201)
      		 
      ==504192==    by 0x522EFD3: start_thread (pthread_create.c:442)

      Reproducible on 11.0+ with at least InnoDB and MyISAM.
      Not reproducible on 10.11.

      Plans are different on 11.0 vs 10.11:

      11.0 (fails)

      		 
      EXPLAIN EXTENDED SELECT * FROM lineitem WHERE l_orderkey IN (248, 250) AND l_linenumber = 3 ORDER BY l_orderkey, l_linenumber;
      		 
      id	select_type	table	type	possible_keys	key	key_len	ref	rows	filtered	Extra
      		 
      1	PRIMARY	<derived3>	ALL	distinct_key	NULL	NULL	NULL	2	100.00	Using filesort
      		 
      1	PRIMARY	lineitem	eq_ref	PRIMARY	PRIMARY	8	tvc_0._col_1,const	1	100.00	
      3	DERIVED	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	No tables used
      		 
      Warnings:
      		 
      Note	1003	/* select#1 */ select `test`.`lineitem`.`l_orderkey` AS `l_orderkey`,`test`.`lineitem`.`l_linenumber` AS `l_linenumber`,`test`.`lineitem`.`l_quantity` AS `l_quantity` from (values (248),(250)) `tvc_0` join `test`.`lineitem` where `test`.`lineitem`.`l_linenumber` = 3 and `test`.`lineitem`.`l_orderkey` = `tvc_0`.`_col_1` order by `test`.`lineitem`.`l_orderkey`

      10.11 (doesn't fail)

      		 
      EXPLAIN EXTENDED SELECT * FROM lineitem WHERE l_orderkey IN (248, 250) AND l_linenumber = 3 ORDER BY l_orderkey, l_linenumber;
      		 
      id	select_type	table	type	possible_keys	key	key_len	ref	rows	filtered	Extra
      		 
      1	PRIMARY	<subquery2>	ALL	distinct_key	NULL	NULL	NULL	2	100.00	Using filesort
      		 
      1	PRIMARY	lineitem	eq_ref	PRIMARY	PRIMARY	8	tvc_0._col_1,const	1	100.00	
      2	MATERIALIZED	<derived3>	ALL	NULL	NULL	NULL	NULL	2	100.00	
      3	DERIVED	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	No tables used
      		 
      Warnings:
      		 
      Note	1003	/* select#1 */ select `test`.`lineitem`.`l_orderkey` AS `l_orderkey`,`test`.`lineitem`.`l_linenumber` AS `l_linenumber`,`test`.`lineitem`.`l_quantity` AS `l_quantity` from `test`.`lineitem` semi join ((values (248),(250)) `tvc_0`) where `test`.`lineitem`.`l_linenumber` = 3 and `test`.`lineitem`.`l_orderkey` = `tvc_0`.`_col_1` order by `test`.`lineitem`.`l_orderkey`

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34922 Assertion `value.length() == FbtImpl::binary_length()' failed in Type_handler_fbt<FbtImpl, TypeCollectionImpl>::Field_fbt::store_native, Assertion `item->null_value' failed in Type_handler::Item_send_str

          • Critical - Crashes, loss of data, severe memory leak.
          • Stalled

          Activity

            Ascending order - Click to sort in descending order
            oleg.smirnov Oleg Smirnov added a comment -

            A proposed solution:

            commit b6259dc137cbf0528620aaf5cdf3f3fb81e58def (HEAD -> bb-11.4-MDEV-32575, origin/bb-11.4-MDEV-32575)
            		 
            Author: Oleg Smirnov <olernov@gmail.com>
            		 
            Date:   Mon Jan 13 17:27:03 2025 +0700
            		 
             
            		 
                MDEV-32575 MSAN / Valgrind errors in test_if_cheaper_ordering upon reaching in_predicate_conversion_threshold
            		 
                
                When converting an IN-list to a subquery, a temporary table stores the IN-list
            		 
                values and participates in join optimization. The problem is the bitmap
            		 
                of usable keys for the temporary table is initialized after the optimization
            		 
                phase, during execution. It happens when the table is opened
            		 
                via `ha_heap::open()`, after the subroutine `set_keys_for_scanning()`
            		 
                is called. Trying to access the bitmap earlier, during optimization, leads
            		 
                to MSAN/Valgrind errors.
            		 
                
                This fix removes the dependency on `set_keys_for_scanning()`. The key bitmap
            		 
                is now dynamically composed on demand in `keys_to_use_for_scanning()`,
            		 
                ensuring correctness without imposing strict call-order constraints.

            sanja, can you please review bb-11.4-MDEV-32575?

            oleg.smirnov Oleg Smirnov added a comment - A proposed solution: commit b6259dc137cbf0528620aaf5cdf3f3fb81e58def (HEAD -> bb-11.4-MDEV-32575, origin/bb-11.4-MDEV-32575) Author: Oleg Smirnov <olernov@gmail.com> Date: Mon Jan 13 17:27:03 2025 +0700   MDEV-32575 MSAN / Valgrind errors in test_if_cheaper_ordering upon reaching in_predicate_conversion_threshold When converting an IN-list to a subquery, a temporary table stores the IN-list values and participates in join optimization. The problem is the bitmap of usable keys for the temporary table is initialized after the optimization phase, during execution. It happens when the table is opened via `ha_heap::open()`, after the subroutine `set_keys_for_scanning()` is called. Trying to access the bitmap earlier, during optimization, leads to MSAN/Valgrind errors. This fix removes the dependency on `set_keys_for_scanning()`. The key bitmap is now dynamically composed on demand in `keys_to_use_for_scanning()`, ensuring correctness without imposing strict call-order constraints. sanja , can you please review bb-11.4- MDEV-32575 ?
            sanja Oleksandr Byelkin added a comment -

            OK to push

            sanja Oleksandr Byelkin added a comment - OK to push
            oleg.smirnov Oleg Smirnov added a comment -

            Pushed to 11.4

            oleg.smirnov Oleg Smirnov added a comment - Pushed to 11.4

            People

              oleg.smirnov Oleg Smirnov
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.