Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32064

Crash when searching for the best split of derived table

    XMLWordPrintable

Details

    Description

      OS: Debian 11
      Server version : MariaDB 10.6.14 (packaged by MariaDB) on tom of ZFS & on test environement MariaDB 10.6.15 (fresh OS install + fresh MariaDB install) without ZFS
      Tested version : MariaDB 10.11.5 packaged by MariaDB (upgrade from 10.6.15)

      Some queries in production generate a segfault.
      Into a lab environement, we can reproduce the segfault, even with an explain of this request.

      Both production & debug environment are using jemalloc.

      Stacktrace for MariaDB 10.6.15 (test server)

      230901  7:21:32 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.15-MariaDB-1:10.6.15+maria~deb11 source revision: 0d16eb35bc981023ce2f4912e8ecde68ca381f4e
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=1
      max_threads=153
      thread_count=1
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467985 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f71c96252d8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f71ea252498 thread_stack 0x30000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x56488f4912ee]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x56488ef56825]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f71eab0e140]
      /usr/sbin/mariadbd(_ZN13st_join_table21choose_best_splittingEjyPK8POSITIONPy+0x574)[0x56488eecf504]
      /usr/sbin/mariadbd(_Z16best_access_pathP4JOINP13st_join_tableyPK8POSITIONjbdPS3_S6_+0x14f)[0x56488ed503ff]
      /usr/sbin/mariadbd(+0x7f41bf)[0x56488ed541bf]
      /usr/sbin/mariadbd(+0x7f447e)[0x56488ed5447e]
      /usr/sbin/mariadbd(+0x7f447e)[0x56488ed5447e]
      /usr/sbin/mariadbd(_Z11choose_planP4JOINy+0x2a4)[0x56488ed54fd4]
      /usr/sbin/mariadbd(+0x81f3a3)[0x56488ed7f3a3]
      /usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x12ec)[0x56488ed8408c]
      /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xa0)[0x56488ed846a0]
      /usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xb3)[0x56488ed84773]
      /usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14f)[0x56488ed84f7f]
      /usr/sbin/mariadbd(+0x7aa641)[0x56488ed0a641]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x46f4)[0x56488ed197b4]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x56488ed1ad67]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x156d)[0x56488ed1d5fd]
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x138)[0x56488ed1f328]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3af)[0x56488ee2e9af]
      /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x56488ee2ecfd]
      /usr/sbin/mariadbd(+0xc43ce2)[0x56488f1a3ce2]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f71eab02ea7]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f71ea709a2f]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f71c967bb30): SELECT test.valdouble as measuredPower,  test.valint1              as csLinkId
      FROM tbl1 test,
      	(SELECT MAX(test.valdate) as maxDate, test.valint1
      	FROM tbl1 test
      	group by test.valint1) as last_entry
      WHERE test.valint1 = last_entry.valint1
      and test.valdate = last_entry.maxDate
      and test.valint1 in  (
      3289475,
      3289496,
      3289562,
      3289593,
      3289594,
      3289595,
      3289626,
      3289642,
      3289658,
      3289739,
      3289750,
      3289761,
      3289762,
      3289763,
      3289774,
      3289780,
      3289951,
      3289957,
      3289988,
      3290034,
      1231562 )
       
      Connection ID (thread ID): 31
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /var/lib/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             15527                15527                processes 
      Max open files            102400               102400               files     
      Max locked memory         524288               524288               bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       15527                15527                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      Kernel version: Linux version 5.10.0-25-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.191-1 (2023-08-16)
      

      Stacktrace for MariaDB 10.11.5 (also test server) :

      230901  7:44:32 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.11.5-MariaDB-1:10.11.5+maria~deb11 source revision: 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=2
      max_threads=153
      thread_count=2
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468037 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f350c00e018
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f352ccc7498 thread_stack 0x30000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55a29f356eee]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55a29ee34365]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f352e06d140]
      /usr/sbin/mariadbd(_ZN13st_join_table21choose_best_splittingEjyPK8POSITIONPy+0x57c)[0x55a29edab55c]
      /usr/sbin/mariadbd(_Z16best_access_pathP4JOINP13st_join_tableyPK8POSITIONjbdPS3_S6_+0x177)[0x55a29ec21087]
      /usr/sbin/mariadbd(+0x85fcc3)[0x55a29ec24cc3]
      /usr/sbin/mariadbd(+0x864b90)[0x55a29ec29b90]
      /usr/sbin/mariadbd(+0x8654b3)[0x55a29ec2a4b3]
      /usr/sbin/mariadbd(+0x8654b3)[0x55a29ec2a4b3]
      /usr/sbin/mariadbd(_Z11choose_planP4JOINy+0x2f4)[0x55a29ec2b0a4]
      /usr/sbin/mariadbd(+0x88c082)[0x55a29ec51082]
      /usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x134c)[0x55a29ec55a5c]
      /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xa0)[0x55a29ec56070]
      /usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xca)[0x55a29ec5615a]
      /usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resulty+0x14f)[0x55a29ec5695f]
      /usr/sbin/mariadbd(+0x807131)[0x55a29ebcc131]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x486e)[0x55a29ebdb2de]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x55a29ebdc987]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15dd)[0x55a29ebdf28d]
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x138)[0x55a29ebe0f28]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3af)[0x55a29ed02e1f]
      /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x55a29ed0316d]
      /usr/sbin/mariadbd(+0xc95022)[0x55a29f05a022]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f352e061ea7]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f352dc68a2f]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f350c042d30): SELECT test.valdouble as measuredPower,  test.valint1              as csLinkId
      FROM tbl1 test,
      	(SELECT MAX(test.valdate) as maxDate, test.valint1
      	FROM tbl1 test
      	group by test.valint1) as last_entry
      WHERE test.valint1 = last_entry.valint1
      and test.valdate = last_entry.maxDate
      and test.valint1 in  (
      3289475,
      3289496,
      3289562,
      3289593,
      3289594,
      3289595,
      3289626,
      3289642,
      3289658,
      3289739,
      3289750,
      3289761,
      3289762,
      3289763,
      3289774,
      3289780,
      3289951,
      3289957,
      3289988,
      3290034,
      1231562 )
       
      Connection ID (thread ID): 57
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /var/lib/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             15527                15527                processes 
      Max open files            102400               102400               files     
      Max locked memory         524288               524288               bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       15527                15527                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      Kernel version: Linux version 5.10.0-25-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.191-1 (2023-08-16)
      

      bdd-test-incident:/tmp# lsb_release -a
      No LSB modules are available.
      Distributor ID:	Debian
      Description:	Debian GNU/Linux 11 (bullseye)
      Release:	11
      Codename:	bullseye
      

      MariaDB [CONF_V3]> show variables like '%malloc%';
      +------------------------+------------------------------------------------------------+
      | Variable_name          | Value                                                      |
      +------------------------+------------------------------------------------------------+
      | version_malloc_library | jemalloc 5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756 |
      +------------------------+------------------------------------------------------------+
      1 row in set (0.002 sec)
       
      MariaDB [CONF_V3]> 
      

      This issue seems to be related to MDEV-31403 - mark as fixed into 10.6.14 .

      Please find attached the small dump file; and a request who crash.

      Usefull information :
      Bug did not occurs when items into the IN are smaller than `in_predicate_conversion_threshold` ;
      ==> Set the `in_predicate_conversion_threshold` to 20 for reproduce it easily.

      Bug did not occurs when we have no indexes on the table (tempoary tests are `create table tbl1 select * from realtable limit 1000000` => no issues)

      In some case (don't know why), first request (source the text file) generate a segfault, second one (after server restart) not. In this case, juste re-import data and crash occurs again.

      Bug occurs with engine InnoDB (as into production) or MyISAM.

      How to reproduce :

      install a fresh MariaDB 10.6.15
      create database test
      use test
      load mysqldump
      set session in_predicate_conversion_threshold=20;
      source request.sql
      

      Attachments

        Issue Links

          Activity

            People

              igor Igor Babaev
              rdem Richard DEMONGEOT
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.