Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29421

Thread (10.6+) and server hangs (10.4/10.5) in 'Opening tables' (on optimized builds) and SIGABRT in safe_mutex_lock (on debug) on I_S read when using Spider

    XMLWordPrintable

Details

    Description

      Please replace '../socket.sock' with the actual socket location.
      The issue is not sporadic. On debug builds, 10.4 to 10.11, we will see the crash below.

      INSTALL PLUGIN Spider SONAME 'ha_spider.so';
      CREATE USER Spider@localhost IDENTIFIED BY 'PWD123';
      CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock', DATABASE 'test', USER 'Spider', PASSWORD 'PWD123');
      CREATE TABLE t (c INT);
      CREATE TABLE t_s (c INT) ENGINE=Spider COMMENT='WRAPPER "mysql", srv "srv", TABLE "t"';
      SET GLOBAL table_open_cache=10;
      CREATE TABLE t1 (a INT) ENGINE=Spider;
      SELECT * FROM t1;
      SELECT * FROM information_schema.tables;
      

      Leads to:

      10.11.0 bc563f1a4b0b38de3b41fd0f0d3d8b7f1aacbd8b (Debug)

      Core was generated by `/test/MD190822-mariadb-10.11.0-linux-x86_64-dbg/bin/mysqld --no-defaults --core'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      [Current thread is 1 (Thread 0x145d84b68700 (LWP 4144754))]
      (gdb) bt
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x0000145d9e0c5859 in __GI_abort () at abort.c:79
      #2  0x000055ccfc7d8b62 in safe_mutex_lock (mp=mp@entry=0x145d84b17ca0 <spider_tbl_mutex>, my_flags=my_flags@entry=0, file=file@entry=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", line=line@entry=5602) at /test/10.11_dbg/mysys/thr_mutex.c:262
      #3  0x0000145d84a623f3 in inline_mysql_mutex_lock (src_line=5602, src_file=0x145d84ae7110 "/test/10.11_dbg/storage/spider/spd_table.cc", that=0x145d84b17ca0 <spider_tbl_mutex>) at /test/10.11_dbg/include/mysql/psi/mysql_thread.h:750
      #4  spider_free_share (share=0x145d3407de58) at /test/10.11_dbg/storage/spider/spd_table.cc:5602
      #5  0x0000145d84a86927 in ha_spider::close (this=0x145d3407c160) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
      #6  0x000055ccfc09ab30 in handler::ha_close (this=0x145d3407c160) at /test/10.11_dbg/sql/handler.cc:3401
      #7  0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table.cc:4534
      #8  0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:225
      #9  0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d3402a068) at /test/10.11_dbg/sql/table_cache.cc:263
      #10 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340dba18) at /test/10.11_dbg/sql/table_cache.cc:363
      #11 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, ot_ctx=ot_ctx@entry=0x145d84b61f00) at /test/10.11_dbg/sql/sql_base.cc:2208
      #12 0x000055ccfbd24307 in open_ltable (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d84b61fe0, lock_type=TL_WRITE, lock_flags=lock_flags@entry=2075) at /test/10.11_dbg/sql/sql_base.cc:5468
      #13 0x0000145d84a01b1d in spider_sys_open_table (thd=thd@entry=0x145d34000db8, tables=tables@entry=0x145d84b61fe0, open_tables_backup=open_tables_backup@entry=0x145d84b62738) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:587
      #14 0x0000145d84a01db0 in spider_open_sys_table (thd=thd@entry=0x145d34000db8, table_name=<optimized out>, table_name@entry=0x145d84ae138a "spider_table_sts", table_name_length=table_name_length@entry=16, write=write@entry=true, open_tables_backup=open_tables_backup@entry=0x145d84b62738, need_lock=need_lock@entry=false, error_num=0x145d84b62734) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:263
      #15 0x0000145d84a0a65e in spider_sys_insert_or_update_table_sts (thd=thd@entry=0x145d34000db8, name=0x145d34013910 "./test/t_s", name_length=10, stat=stat@entry=0x145d3411e620, need_lock=need_lock@entry=false) at /test/10.11_dbg/storage/spider/spd_sys_table.cc:3240
      #16 0x0000145d84a625c2 in spider_free_share (share=0x145d3411dde8) at /test/10.11_dbg/storage/spider/spd_table.cc:5633
      #17 0x0000145d84a86927 in ha_spider::close (this=0x145d3411d230) at /test/10.11_dbg/storage/spider/ha_spider.cc:577
      #18 0x000055ccfc09ab30 in handler::ha_close (this=0x145d3411d230) at /test/10.11_dbg/sql/handler.cc:3401
      #19 0x000055ccfbec25a1 in closefrm (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table.cc:4534
      #20 0x000055ccfbfeec17 in intern_close_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:225
      #21 0x000055ccfbfeed0a in tc_remove_table (table=table@entry=0x145d34097dc8) at /test/10.11_dbg/sql/table_cache.cc:263
      #22 0x000055ccfbfeee9c in tc_add_table (thd=thd@entry=0x145d34000db8, table=0x145d340b6a88) at /test/10.11_dbg/sql/table_cache.cc:363
      #23 0x000055ccfbd21a83 in open_table (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, ot_ctx=ot_ctx@entry=0x145d84b62fa0) at /test/10.11_dbg/sql/sql_base.cc:2208
      #24 0x000055ccfbd261b5 in open_and_process_table (ot_ctx=0x145d84b62fa0, has_prelocking_list=false, prelocking_strategy=0x145d84b63038, flags=1090, counter=0x145d84b63034, tables=0x145d3419c230, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.cc:4106
      #25 open_tables (thd=thd@entry=0x145d34000db8, options=@0x145d84b64758: {m_options = DDL_options_st::OPT_NONE}, start=start@entry=0x145d84b63028, counter=counter@entry=0x145d84b63034, flags=1090, prelocking_strategy=prelocking_strategy@entry=0x145d84b63038) at /test/10.11_dbg/sql/sql_base.cc:4593
      #26 0x000055ccfbd2732d in open_tables (prelocking_strategy=0x145d84b63038, flags=<optimized out>, counter=0x145d84b63034, tables=0x145d84b63028, thd=0x145d34000db8) at /test/10.11_dbg/sql/sql_base.h:266
      #27 open_normal_and_derived_tables (thd=thd@entry=0x145d34000db8, tables=<optimized out>, tables@entry=0x145d3419c230, flags=<optimized out>, dt_phases=dt_phases@entry=3) at /test/10.11_dbg/sql/sql_base.cc:5631
      #28 0x000055ccfbd27420 in open_tables_only_view_structure (thd=thd@entry=0x145d34000db8, table_list=table_list@entry=0x145d3419c230, can_deadlock=can_deadlock@entry=false) at /test/10.11_dbg/sql/sql_base.cc:5681
      #29 0x000055ccfbe41f0c in fill_schema_table_by_open (thd=thd@entry=0x145d34000db8, mem_root=mem_root@entry=0x145d84b64f30, is_show_fields_or_keys=is_show_fields_or_keys@entry=false, table=table@entry=0x145d341026d0, schema_table=schema_table@entry=0x55ccfd26dc80 <schema_tables+2368>, orig_db_name=orig_db_name@entry=0x145d34111f58, orig_table_name=0x145d34112198, open_tables_state_backup=0x145d84b64f70, can_deadlock=false) at /test/10.11_dbg/sql/sql_show.cc:4718
      #30 0x000055ccfbe6573e in get_all_tables (thd=0x145d34000db8, tables=0x145d340142d8, cond=<optimized out>) at /test/10.11_dbg/sql/sql_show.cc:5389
      #31 0x000055ccfbe66cd0 in get_schema_tables_result (join=join@entry=0x145d34015e28, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /test/10.11_dbg/sql/sql_show.cc:8943
      #32 0x000055ccfbe3b1ed in JOIN::exec_inner (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_string.h:687
      #33 0x000055ccfbe3be28 in JOIN::exec (this=this@entry=0x145d34015e28) at /test/10.11_dbg/sql/sql_select.cc:4590
      #34 0x000055ccfbe39bac in mysql_select (thd=thd@entry=0x145d34000db8, tables=0x145d340142d8, fields=@0x145d34013f88: {<base_list> = {<Sql_alloc> = {<No data fields>}, first = 0x145d34014278, last = 0x145d3410f658, elements = 23}, <No data fields>}, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2701396736, result=0x145d34015e00, unit=0x145d34004ff0, select_lex=0x145d34013ce8) at /test/10.11_dbg/sql/sql_select.cc:5070
      #35 0x000055ccfbe3a3a2 in handle_select (thd=thd@entry=0x145d34000db8, lex=lex@entry=0x145d34004f18, result=result@entry=0x145d34015e00, setup_tables_done_option=setup_tables_done_option@entry=0) at /test/10.11_dbg/sql/sql_select.cc:581
      #36 0x000055ccfbda45a6 in execute_sqlcom_select (thd=thd@entry=0x145d34000db8, all_tables=0x145d340142d8) at /test/10.11_dbg/sql/sql_parse.cc:6261
      #37 0x000055ccfbdb08c7 in mysql_execute_command (thd=thd@entry=0x145d34000db8, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false) at /test/10.11_dbg/sql/sql_parse.cc:3945
      #38 0x000055ccfbd9e882 in mysql_parse (thd=thd@entry=0x145d34000db8, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x145d84b67330) at /test/10.11_dbg/sql/sql_parse.cc:8035
      #39 0x000055ccfbdabe6a in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x145d34000db8, packet=packet@entry=0x145d3400b6e9 "SELECT * FROM information_schema.tables", packet_length=packet_length@entry=39, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_class.h:1339
      #40 0x000055ccfbdae574 in do_command (thd=0x145d34000db8, blocking=blocking@entry=true) at /test/10.11_dbg/sql/sql_parse.cc:1407
      #41 0x000055ccfbf101da in do_handle_one_connection (connect=<optimized out>, connect@entry=0x55ccfedb1628, put_in_cache=put_in_cache@entry=true) at /test/10.11_dbg/sql/sql_connect.cc:1418
      #42 0x000055ccfbf106e3 in handle_one_connection (arg=0x55ccfedb1628) at /test/10.11_dbg/sql/sql_connect.cc:1312
      #43 0x0000145d9e5d6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #44 0x0000145d9e1c2133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      Optimized builds will thread-hang with:

      10.10.2 87e8463e0454a04c2bbaa38d44227c491fb07dc1 (Optimized)

      MariaDB [test]> SHOW PROCESSLIST;
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
      | Id | User        | Host      | db   | Command | Time | State                                              | Info                                    | Progress |
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
      |  4 | root        | localhost | test | Query   | 1289 | Opening tables                                     | SELECT * FROM information_schema.tables |    0.000 |
      |  5 | system user |           | NULL | Sleep   | 1289 | Reset for next command                             | NULL                                    |    0.000 |
      |  6 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      |  7 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      |  8 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 11 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 12 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 14 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 15 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 16 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 18 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      |  9 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 10 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 13 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 17 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                                    |    0.000 |
      | 19 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 20 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 21 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 22 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 23 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 24 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                                    |    0.000 |
      | 25 | Spider      | localhost | test | Sleep   | 1288 |                                                    | NULL                                    |    0.000 |
      | 26 | root        | localhost | test | Query   |    0 | starting                                           | SHOW PROCESSLIST                        |    0.000 |
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-----------------------------------------+----------+
      23 rows in set (0.001 sec)
      

      A shutdown is still possible for 10.6 to 10.11 in this case. However, 10.4 and 10.5 will fully hang during shutdown.

      Bug confirmed present in:
      MariaDB: 10.4.27 (dbg), 10.4.27 (opt), 10.5.18 (dbg), 10.5.18 (opt), 10.6.10 (dbg), 10.6.10 (opt), 10.7.6 (dbg), 10.7.6 (opt), 10.8.5 (dbg), 10.8.5 (opt), 10.9.2 (dbg), 10.9.2 (opt), 10.10.2 (dbg), 10.10.2 (opt), 10.11.0 (dbg), 10.11.0 (opt)

      Bug (or feature/syntax) confirmed not present in:
      MySQL: 5.5.62 (dbg), 5.5.62 (opt), 5.6.51 (dbg), 5.6.51 (opt), 5.7.38 (dbg), 5.7.38 (opt), 8.0.29 (dbg), 8.0.29 (opt)

      SIGABRT|safe_mutex_lock|inline_mysql_mutex_lock|spider_free_share|ha_spider::close
      

      Attachments

        Issue Links

          Activity

            People

              ycp Yuchen Pei
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.