Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33510

Server crash : mysqld got signal 11 in st_join_table::save_explain_data

    XMLWordPrintable

Details

    • Bug
    • Status: In Progress (View Workflow)
    • Critical
    • Resolution: Unresolved
    • 10.6.16
    • 10.6
    • None
    • Red Hat Enterprise Linux Server release 7.9 (Maipo)
      16 CPU
      150 GB

    Description

      Hi,

      I can't get my head around the cause for a crash we've had recently.
      It only happened once, and I couldn't replicate the issue. Could you help me understand the backtrace ?

      I have to mention that we upgraded from 10.1.38 to 10.6.16 recently (late January). We never had such a crash before.

      240216 10:55:38 [ERROR] mysqld got signal 11 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.6.16-MariaDB-log source revision: b83c379420a8846ae4b28768d3c81fa354cca056
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=268
      max_threads=4002
      thread_count=233
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4846808 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f258c4a9588
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f27217bac40 thread_stack 0x49000
      Can't start addr2line
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55605f839eae]
      /usr/sbin/mariadbd(handle_fatal_signal+0x2f7)[0x55605f28eac7]
      /lib64/libpthread.so.0(+0xf630)[0x7f401d447630]
      /usr/sbin/mariadbd(_ZN13st_join_table17save_explain_dataEP20Explain_table_accessybPS_+0x30b)[0x55605f0a926b]
      /usr/sbin/mariadbd(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0xd6f)[0x55605f0aaf1f]
      /usr/sbin/mariadbd(_ZN4JOIN17save_explain_dataEP13Explain_querybbbb+0x94)[0x55605f0ab084]
      /usr/sbin/mariadbd(_ZN4JOIN13build_explainEv+0x7c)[0x55605f0ab2dc]
      /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0x71)[0x55605f0c2751]
      /usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xb8)[0x55605f0c2868]
      /usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1e4)[0x55605f0c3174]
      /usr/sbin/mariadbd(+0x67bd11)[0x55605ef12d11]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x34a3)[0x55605f0610c3]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x20b)[0x55605f06346b]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x14a0)[0x55605f0658a0]
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x123)[0x55605f066dc3]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x187)[0x55605f164cd7]
      /usr/sbin/mariadbd(handle_one_connection+0x34)[0x55605f164f74]
      /usr/sbin/mariadbd(+0xc466ac)[0x55605f4dd6ac]
      /lib64/libpthread.so.0(+0x7ea5)[0x7f401d43fea5]
      /lib64/libc.so.6(clone+0x6d)[0x7f401c95ab0d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f258c851880): SELECT sys_ui_ng_action0.`sys_overrides` FROM (sys_ui_ng_action sys_ui_ng_action0  INNER JOIN sys_metadata sys_metadata0 ON sys_ui_ng_action0.`sys_id` = sys_metadata0.`sys_id` )  WHERE sys_ui_ng_action0.`table` IN ('sys_user' , 'global') AND sys_ui_ng_action0.`active` = 1 AND sys_ui_ng_action0.`list_button` = 1 AND (sys_ui_ng_action0.`show_for` = 'any' OR sys_ui_ng_action0.`show_for` = 'update') AND sys_ui_ng_action0.`sys_overrides` IS NOT NULL  AND sys_ui_ng_action0.`sys_domain` != 'global' AND (sys_ui_ng_action0.`sys_domain_path` = '/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/!!!/!!#/!!!/!!$/!!)/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/!!!/!!#/!!!/!!$/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/!!!/!!#/!!!/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/!!!/!!#/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/!!!/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/!!#/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/!!!/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/!!!/' OR sys_ui_ng_action0.`sys_domain_path` = '!!!/' OR sys_ui_ng_action0.`sys_domain_path` = '/') /* serv1, gs:F6CC5A87FD108A50DB4614677409CE67, tx:a60e560bfd108a50db4614677409ce93, hash:-1250232669 */
       
      Connection ID (thread ID): 42129
      Status: KILL_QUERY
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /basedata00/INSTANCE_NAME/data
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        unlimited            unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             611711               611711               processes 
      Max open files            1048576              1048576              files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       611711               611711               signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      Kernel version: Linux version 3.10.0-1160.81.1.el7.x86_64 (mockbuild@x86-vm-38.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Thu Nov 24 12:21:22 UTC 2022
      

      Demangled relevant stack:

      /usr/sbin/mariadbd(st_join_table::save_explain_data(Explain_table_access*, unsigned long long, bool, st_join_table*)+0x30b)[0x55605f0a926b]
      /usr/sbin/mariadbd(JOIN::save_explain_data_intern(Explain_query*, bool, bool, bool, char const*)+0xd6f)[0x55605f0aaf1f]
      /usr/sbin/mariadbd(JOIN::save_explain_data(Explain_query*, bool, bool, bool, bool)+0x94)[0x55605f0ab084]
      /usr/sbin/mariadbd(JOIN::build_explain()+0x7c)[0x55605f0ab2dc]
      /usr/sbin/mariadbd(JOIN::optimize()+0x71)[0x55605f0c2751]
      /usr/sbin/mariadbd(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0xb8)[0x55605f0c2868]
      /usr/sbin/mariadbd(handle_select(THD*, LEX*, select_result*, unsigned long)+0x1e4)[0x55605f0c3174]
      /usr/sbin/mariadbd(+0x67bd11)[0x55605ef12d11]
      /usr/sbin/mariadbd(mysql_execute_command(THD*, bool)+0x34a3)[0x55605f0610c3]
      

      Here is the cnf file :

      #
      # These groups are read by MariaDB server.
      # Use it for options that only the server (but not clients) should see
      #
      # See the examples of server my.cnf files in /usr/share/mysql/
      #
       
      # this is read by the standalone daemon and embedded servers
      [server]
       
      # this is only for the mysqld standalone daemon
      [mysqld]
       
      server-id                       = 1
      log_bin                         = /base/INSTANCE_NAME/arc/mysql-bin
      skip_name_resolve               = 1
      skip-slave-start
      tmpdir                          = /base/INSTANCE_NAME/data
      log_slave_updates               = 1
      max_connect_errors              = 20000 # 100 default
      binlog_format                   = MIXED # Mixed default
       
      basedir                         = /usr
       
      port                            = 3306
      socket                          = /base/INSTANCE_NAME/conf/mysql_INSTANCE_NAME.sock
       
      datadir                         = /base/INSTANCE_NAME/data
      innodb_tmpdir                   = /base/INSTANCE_NAME/data
       
      general_log                     = off
      general_log_file                = /base/INSTANCE_NAME/log/mariadb_INSTANCE_NAME_gal.log
      log-error                       = /base/INSTANCE_NAME/log/mariadb_INSTANCE_NAME.log
       
      secure_auth                     = ON
       
      key_buffer_size                 = 128M  
      max_allowed_packet              = 32M
      tmp_table_size                  = 64M 
      max_heap_table_size             = 64M
      sort_buffer_size                = 1M 
      myisam_sort_buffer_size         = 256K
       
      net_write_timeout               = 900 
      net_read_timeout                = 900 
       
      sync_binlog                     = 0
       
      # Slow query conf 
      slow_query_log                  = ON
      long_query_time                 = 10
      slow_query_log_file            = /base/INSTANCE_NAME/log/slow_queries_INSTANCE_NAME.log
       
       
      expire_logs_days                = 3
      #log-slave-updates               = 1
      #read-only                       = 0
      #binlog_cache_size               = 1M
      slave_compressed_protocol       = 1
      #log-bin-trust-function-creators = 1
       
      default_storage_engine          = innodb
      character_set_server            = utf8
       
      query_cache_size                = 0
      query_cache_type                = 0
      innodb_buffer_pool_size         = 96G
      innodb_io_capacity              = 65535
      innodb_open_files               = 8000
      open_files_limit                = 37535
      table_open_cache                = 18000
      table_definition_cache          = 12000
       
      max_connections                 = 4000
      innodb_lock_wait_timeout        = 50
      innodb_sync_spin_loops          = 20
      innodb_log_file_size            = 512M
      innodb_read_io_threads          = 32
      innodb_write_io_threads         = 32
      innodb_autoinc_lock_mode        = 1
      innodb_purge_threads            = 4
      innodb_purge_batch_size         = 300
      innodb_max_purge_lag            = 5000000
      thread_cache_size         	= 128
       
      back_log                        = 2500
       
      innodb_log_buffer_size          = 64M
      innodb_page_size                = 16K
      join_buffer_size                = 256K
      innodb_strict_mode              = off
       
      innodb_use_native_aio           = 1
      innodb_stats_persistent         = 1
      innodb_spin_wait_delay          = 48
       
      innodb_adaptive_flushing        = 1
      innodb_checksum_algorithm       = crc32
       
      #optimizer_switch               = index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=off,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
       
      innodb_flush_log_at_trx_commit  = 1
      innodb_stats_on_metadata        = 0
      innodb_flush_method             = O_DIRECT
      innodb_file_per_table           = on
       
      innodb_data_file_path           = ibdata1:10M:autoextend 
       
      innodb_monitor_disable          = ALL
      performance_schema              = on
       
      local-infile                    = off
       
      bind-address                    = 0.0.0.0
       
      symbolic-links                  = 0
      lock_wait_timeout               = 14400
       
      # Upgrade MariaDB 10.6.16
      sql_mode                        = NO_ENGINE_SUBSTITUTION
      slave_parallel_max_queued       = 524288
      innodb_print_all_deadlocks      = on
      innodb_max_dirty_pages_pct      = 75
      slave_parallel_mode             = conservative
      innodb_adaptive_hash_index      = on
      log_queries_not_using_indexes   = 0
      log_bin_trust_function_creators = 1
      slave_parallel_threads          = 16
      eq_range_index_dive_limit       = 0
       
      #innodb_log_files_in_group       = 3 # Removed from 10.6
      #innodb_support_xa               = 1 # Removed from 10.3.0
      #innodb_thread_concurrency       = 0 # Removed from 10.6
      #innodb_buffer_pool_instances    = 12  # Removed from 10.6
      #innodb_additional_mem_pool_size = 20M # Removed from 10.2
      #innodb_file_format              = Barracuda # Removed from 10.6
       
       
       
       
      #
      # * Galera-related settings
      #
      [galera]
      # Mandatory settings
      #wsrep_on                       = ON
      #wsrep_provider                 =
      #wsrep_cluster_address          =
      #binlog_format                  = row
      #default_storage_engine         = InnoDB
      #innodb_autoinc_lock_mode       = 2
      #
      # Allow server to accept connections on all interfaces.
      #
      #bind-address                   = 0.0.0.0
      #
      # Optional setting
      #wsrep_slave_threads            = 1
      #innodb_flush_log_at_trx_commit	= 0
       
      # this is only for embedded server
      [embedded]
       
      # This group is only read by MariaDB servers, not by MySQL.
      # If you use the same .cnf file for MySQL and MariaDB,
      # you can put MariaDB-only options here
      [mariadb]
       
      # This group is only read by MariaDB-10.2 servers.
      # If you use the same .cnf file for MariaDB of different versions,
      # use this group for options that older servers don't understand
      [mariadb-10.2]
      #enforce_gtid_consistency=1
       
      #Erreur reverse lookup
      skip_name_resolve               = 1
       
      [mysql]
      no-auto-rehash
      

      Attachments

        Issue Links

          Activity

            People

              igor Igor Babaev
              JulienD Julien D
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.