Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15687

Frequent table partition meta data queries cause mysqld to crash periodically

    XMLWordPrintable

    Details

      Description

      In our system, one of our software component(called 'agent') sends this query every 3 seconds to mysqld:

      select SUBPARTITION_NAME from  information_schema.partitions where TABLE_NAME='gtid_log_t' and SUBPARTITION_ORDINAL_POSITION=3;
      

      and about every 15 minutes, mysqld crashes at below call stack (the same call stack at every crash) and the thd->query_string is always the above stmt at every crash. and when the 'agent' is turned off(no longer send any stmt to mysqld), mysqld doesn't crash.

      I am happy to provide any configuration variables if you need any.

      Hope you can fix it, thank you!

      #0  lf_pinbox_real_free (pins=0x260e500) at /data/home/jenkins/workspace/mariadb/mariadb/mysys/lf_alloc-pin.c:355
      #1  0x0000000000c4eeb5 in l_delete (pins=0x260e500, keylen=16, key=0x7f7ab003e6a1 "test", hashnr=2685431289, cs=0x15ee820 <my_charset_bin>, head=0x2616448) at /data/home/jenkins/workspace/mariadb/mariadb/mysys/lf_hash.c:245
      #2  lf_hash_delete (hash=0x1687540 <tdc_hash>, pins=0x260e500, key=0x7f7ab003e6a1, keylen=16) at /data/home/jenkins/workspace/mariadb/mariadb/mysys/lf_hash.c:453
      #3  0x000000000075ff3f in tdc_acquire_share (thd=0x5283b0a8, db=0x7f7ab000aa58 "test", table_name=0x7f7ab000aa60 "gtid_log_t", key=0x7f7ab003e6a1 "test", key_length=16, hash_value=532023301, flags=1, out_table=0x7f7b7c7afcb0)
          at /data/home/jenkins/workspace/mariadb/mariadb/sql/table_cache.cc:644
      #4  0x00000000005c2502 in open_table (thd=0x5283b0a8, table_list=0x7f7ab003e260, ot_ctx=0x7f7b7c7afd60) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_base.cc:2419
      #5  0x00000000005c336e in open_and_process_table (ot_ctx=0x7f7b7c7afd60, has_prelocking_list=false, prelocking_strategy=0x7f7b7c7afe70, flags=1026, counter=0x7f7b7c7afe7c, tables=0x7f7ab003e260, lex=0x7f7b7c7aff00, thd=0x5283b0a8)
          at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_base.cc:4068
      #6  open_tables (thd=0x5283b0a8, options=..., start=0x7f7b7c7afe58, counter=0x7f7b7c7afe7c, flags=1026, prelocking_strategy=0x7f7b7c7afe70) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_base.cc:4579
      #7  0x00000000005c3ee3 in open_tables (prelocking_strategy=0x7f7b7c7afe70, flags=<optimized out>, counter=0x7f7b7c7afe7c, tables=0x7f7b7c7afe58, thd=0x5283b0a8) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_base.h:246
      #8  open_normal_and_derived_tables (thd=0x5283b0a8, tables=0x7f7ab003e260, flags=<optimized out>, dt_phases=34) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_base.cc:5304
      #9  0x000000000066abf5 in fill_schema_table_by_open (thd=0x5283b0a8, is_show_fields_or_keys=false, table=0x7f7ab0038360, schema_table=0x15c2660 <schema_tables+1152>, orig_db_name=0x7f7ab000a780, orig_table_name=0x7f7ab0009e68, 
          open_tables_state_backup=0x7f7b7c7b2f00, can_deadlock=false) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_show.cc:4110
      #10 0x000000000066e4fe in get_all_tables (thd=0x5283b0a8, tables=0x7f7ab0004f88, cond=<optimized out>) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_show.cc:4735
      #11 0x0000000000668447 in get_schema_tables_result (join=0x7f7ab0007a28, executed_place=PROCESSED_BY_JOIN_EXEC) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_show.cc:8052
      #12 0x000000000065fe5d in JOIN::exec_inner (this=0x7f7ab0007a28) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_select.cc:2670
      #13 0x000000000065ead7 in exec (this=0x7f7ab0007a28) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_select.cc:2519
      #14 mysql_select (thd=0x5283b0a8, rref_pointer_array=0x5283f310, tables=<optimized out>, wild_num=0, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2684619520, 
          result=0x7f7ab0007a08, unit=0x5283e9a8, select_lex=0x5283f098) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_select.cc:3509
      #15 0x0000000000662106 in handle_select (thd=0x5283b0a8, lex=0x5283e8e0, result=0x7f7ab0007a08, setup_tables_done_option=0) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_select.cc:385
      #16 0x00000000005ff561 in execute_sqlcom_select (thd=0x5283b0a8, all_tables=0x7f7ab0004f88) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_parse.cc:6232
      #17 0x000000000060a3f0 in mysql_execute_command (thd=0x5283b0a8) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_parse.cc:3156
      #18 0x000000000060cddd in mysql_parse (thd=0x5283b0a8, rawbuf=<optimized out>, length=126, parser_state=0x7f7b7c7b6860) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_parse.cc:7632
      #19 0x000000000060fb90 in dispatch_command (command=COM_QUERY, thd=0x5283b0a8, packet=0x52840cd9 "select SUBPARTITION_NAME from  information_schema.partitions where TABLE_NAME='gtid_log_t' and SUBPARTITION_ORDINAL_POSITION=3", 
          packet_length=126) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_parse.cc:1516
      #20 0x000000000060ff83 in do_command (thd=0x5283b0a8) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_parse.cc:1130
      #21 0x00000000006e0b2c in do_handle_one_connection (thd_arg=<optimized out>) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_connect.cc:1349
      #22 0x00000000006e0c70 in handle_one_connection (arg=0x5283b0a8) at /data/home/jenkins/workspace/mariadb/mariadb/sql/sql_connect.cc:1261
      #23 0x00007f8824225dc5 in start_thread () from /lib64/libpthread.so.0
      #24 0x00007f8822cd874d in clone () from /lib64/libc.so.
      

      (gdb) f 0
      #0  lf_pinbox_real_free (pins=0x260e500) at /data/home/jenkins/workspace/mariadb/mariadb/mysys/lf_alloc-pin.c:355
      355     in /data/home/jenkins/workspace/mariadb/mariadb/mysys/lf_alloc-pin.c
      (gdb) f 3
      #3  0x000000000075ff3f in tdc_acquire_share (thd=0x5283b0a8, db=0x7f7ab000aa58 "test", table_name=0x7f7ab000aa60 "gtid_log_t", key=0x7f7ab003e6a1 "test", key_length=16, hash_value=532023301, flags=1, out_table=0x7f7b7c7afcb0)
          at /data/home/jenkins/workspace/mariadb/mariadb/sql/table_cache.cc:644
      644     /data/home/jenkins/workspace/mariadb/mariadb/sql/table_cache.cc: No such file or directory.
      (gdb) p thd->query_string 
      $1 = {string = {str = 0x7f7ab0004ca0 "select SUBPARTITION_NAME from  information_schema.partitions where TABLE_NAME='gtid_log_t' and SUBPARTITION_ORDINAL_POSITION=3", length = 126}, cs = 0x164ea60 <my_charset_utf8_general_ci>}
      (gdb) f 3
      #3  0x000000000075ff3f in tdc_acquire_share (thd=0x5283b0a8, db=0x7f7ab000aa58 "test", table_name=0x7f7ab000aa60 "gtid_log_t", key=0x7f7ab003e6a1 "test", key_length=16, hash_value=532023301, flags=1, out_table=0x7f7b7c7afcb0)
          at /data/home/jenkins/workspace/mariadb/mariadb/sql/table_cache.cc:644
      

      And later I found that the select stmt at top of this report, is not the root cause of the crash — I turned off the sending of that stmt, and mysqld crashes at another call stack and the same line of code, as the attached picture(crash2.png) shows. and this new stmt is also sent by our 'agent' component.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              serg Sergei Golubchik
              Reporter:
              DZW David Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration