Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34588

Assertion `thd == _current_thd() || ((&(&thd->LOCK_thd_data)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&thd->LOCK_thd_data)->m_mutex)->thread))' failed in thd_get_ha_data and thread hangs

Details

    Description

      INSTALL PLUGIN Spider SONAME 'ha_spider.so';
      CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock',DATABASE'',USER'',PASSWORD'');
      CREATE TABLE t1 (c INT,KEY(c)) ENGINE=Spider PARTITION BY KEY(c) PARTITIONS 1;
      CREATE TABLE t2 (c INT,d INT) ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "srv",TABLE "t1"';
      LOCK TABLES t2 READ,t1 WRITE;
      

      Leads to:

      11.2.5 a21e49cbcc5f4adb1a1b4970ceead6a85e968063 (Debug)

      mariadbd: /test/11.2_dbg/sql/sql_class.cc:455: void* thd_get_ha_data(const THD*, const handlerton*): Assertion `thd == _current_thd() || ((&(&thd->LOCK_thd_data)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&thd->LOCK_thd_data)->m_mutex)->thread))' failed.
      

      11.2.5 a21e49cbcc5f4adb1a1b4970ceead6a85e968063 (Debug)

      Core was generated by `/test/MD190624-mariadb-11.2.5-linux-x86_64-dbg/bin/mariadbd --no-defaults --max'.
      Program terminated with signal SIGABRT, Aborted.
      #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
      Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
      [Current thread is 1 (LWP 3996738)]
      (gdb) bt
      #0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:44
      #1  __pthread_kill_internal (signo=6, threadid=<optimized out>)at ./nptl/pthread_kill.c:78
      #2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6)at ./nptl/pthread_kill.c:89
      #3  0x000015229da42866 in __GI_raise (sig=sig@entry=6)at ../sysdeps/posix/raise.c:26
      #4  0x000015229da268b7 in __GI_abort () at ./stdlib/abort.c:79
      #5  0x000015229da267db in __assert_fail_base (fmt=0x15229dbc5168 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x5643e79cddb0 "thd == _current_thd() || ((&(&thd->LOCK_thd_data)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&thd->LOCK_thd_data)->m_mutex)->thread))", file=file@entry=0x5643e79cdc88 "/test/11.2_dbg/sql/sql_class.cc", line=line@entry=455, function=function@entry=0x5643e79cdd78 "void* thd_get_ha_data(const THD*, const handlerton*)") at ./assert/assert.c:92
      #6  0x000015229da39186 in __assert_fail (assertion=0x5643e79cddb0 "thd == _current_thd() || ((&(&thd->LOCK_thd_data)->m_mutex)->count > 0 && pthread_equal(pthread_self(), (&(&thd->LOCK_thd_data)->m_mutex)->thread))", file=0x5643e79cdc88 "/test/11.2_dbg/sql/sql_class.cc", line=455, function=0x5643e79cdd78 "void* thd_get_ha_data(const THD*, const handlerton*)") at ./assert/assert.c:101
      #7  0x00005643e6c4251e in thd_get_ha_data (thd=0x152248000d58, hton=0x15224c03d318) at /test/11.2_dbg/sql/sql_class.cc:455
      #8  0x00001522880d640b in spider_get_trx (thd=thd@entry=0x152248000d58, regist_allocated_thds=regist_allocated_thds@entry=true, error_num=error_num@entry=0x15229c06ba14)at /test/11.2_dbg/storage/spider/spd_trx.cc:1141
      #9  0x00001522880d77d5 in spider_check_trx_and_get_conn (thd=0x152248000d58, spider=spider@entry=0x15224803ace0)at /test/11.2_dbg/storage/spider/spd_trx.cc:3396
      #10 0x0000152288141fe9 in ha_spider::append_lock_tables_list (this=0x15224803ace0) at /test/11.2_dbg/storage/spider/ha_spider.cc:11824
      #11 0x0000152288142643 in ha_spider::store_lock (this=0x15224803ace0, thd=<optimized out>, to=0x15224c2918e0, lock_type=TL_WRITE)at /test/11.2_dbg/storage/spider/ha_spider.cc:760
      #12 0x00005643e725b7bb in ha_partition::store_lock (this=0x15224803a400, thd=0x15224c000d58, to=0x15224c2918e0, lock_type=TL_WRITE)at /test/11.2_dbg/sql/ha_partition.cc:4412
      #13 0x00005643e712eb4f in get_lock_data (thd=thd@entry=0x15224c000d58, table_ptr=table_ptr@entry=0x15224c014728, count=count@entry=2, flags=flags@entry=1) at /test/11.2_dbg/sql/lock.cc:825
      #14 0x00005643e712ef7a in mysql_lock_tables (thd=thd@entry=0x15224c000d58, tables=tables@entry=0x15224c014728, count=count@entry=2, flags=flags@entry=0) at /test/11.2_dbg/sql/lock.cc:301
      #15 0x00005643e6c3023f in lock_tables (thd=thd@entry=0x15224c000d58, tables=0x15224c013708, count=<optimized out>, flags=flags@entry=0)at /test/11.2_dbg/sql/sql_base.cc:5916
      #16 0x00005643e6ca77f5 in lock_tables_open_and_lock_tables (thd=thd@entry=0x15224c000d58, tables=<optimized out>)at /test/11.2_dbg/sql/sql_parse.cc:2999
      #17 0x00005643e6cb2f01 in mysql_execute_command (thd=thd@entry=0x15224c000d58, is_called_from_prepared_stmt=is_called_from_prepared_stmt@entry=false)at /test/11.2_dbg/sql/sql_parse.cc:4959
      #18 0x00005643e6cb7010 in mysql_parse (thd=thd@entry=0x15224c000d58, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x15229c06c2e0)at /test/11.2_dbg/sql/sql_parse.cc:7920
      #19 0x00005643e6cb93d3 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x15224c000d58, packet=packet@entry=0x15224c00b2f9 "LOCK TABLES t2 READ,t1 WRITE", packet_length=packet_length@entry=28, blocking=blocking@entry=true)at /test/11.2_dbg/sql/sql_class.h:247
      #20 0x00005643e6cbb76c in do_command (thd=0x15224c000d58, blocking=blocking@entry=true) at /test/11.2_dbg/sql/sql_parse.cc:1407
      #21 0x00005643e6e22c49 in do_handle_one_connection (connect=<optimized out>, connect@entry=0x5643ea69d398, put_in_cache=put_in_cache@entry=true)at /test/11.2_dbg/sql/sql_connect.cc:1439
      #22 0x00005643e6e22f3e in handle_one_connection (arg=arg@entry=0x5643ea69d398)at /test/11.2_dbg/sql/sql_connect.cc:1341
      #23 0x00005643e727552c in pfs_spawn_thread (arg=0x5643ea5fc5e8)at /test/11.2_dbg/storage/perfschema/pfs.cc:2201
      #24 0x000015229da97ada in start_thread (arg=<optimized out>)at ./nptl/pthread_create.c:444
      #25 0x000015229db2847c in clone3 ()at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
      

      In optimized builds, the testcase hangs the client thread.

      In 10.5 debug, the testcase also hangs. In 10.6+ debug the testcase asserts with the stack above.

      Here is another testcase which hangs on all versions in all build types:

      INSTALL PLUGIN Spider SONAME 'ha_spider.so';
      CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET '../socket.sock',DATABASE'',USER'',PASSWORD'');
      CREATE TABLE t1 (c INT,KEY(c)) ENGINE=Spider;
      CREATE TABLE t2 (c INT,d INT) ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "srv",TABLE "t1"';
      LOCK TABLES t2 READ,t1 WRITE;
      

      Attachments

        Issue Links

          Activity

            Roel Roel Van de Paar added a comment - - edited

            MTR testcase for first testcase (hangs 10.5 dbg+opt, asserts 10.6+ debug):

            --source include/have_partition.inc
            --let $SOCKET= `SELECT @@global.socket`
            INSTALL PLUGIN Spider SONAME 'ha_spider.so';
            CREATE USER spider@localhost IDENTIFIED BY 'pwd';
            GRANT ALL ON test.* TO spider@localhost;
            eval CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET "$SOCKET",DATABASE 'test',USER 'spider',PASSWORD 'pwd');
            SET spider_same_server_link=1;
            CREATE TABLE t1 (c INT,KEY(c)) ENGINE=Spider PARTITION BY KEY(c) PARTITIONS 1;
            CREATE TABLE t2 (c INT,d INT) ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "srv",TABLE "t1"';
            LOCK TABLES t2 READ,t1 WRITE;
            

            And for second testcase (hangs always):

            --let $SOCKET= `SELECT @@global.socket`
            INSTALL PLUGIN Spider SONAME 'ha_spider.so';
            CREATE USER spider@localhost IDENTIFIED BY 'pwd';
            GRANT ALL ON test.* TO spider@localhost;
            eval CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET "$SOCKET",DATABASE 'test',USER 'spider',PASSWORD 'pwd');
            SET spider_same_server_link=1;
            CREATE TABLE t1 (c INT,KEY(c)) ENGINE=Spider;
            CREATE TABLE t2 (c INT,d INT) ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "srv",TABLE "t1"';
            LOCK TABLES t2 READ,t1 WRITE;
            

            Roel Roel Van de Paar added a comment - - edited MTR testcase for first testcase (hangs 10.5 dbg+opt, asserts 10.6+ debug): --source include/have_partition.inc --let $SOCKET= `SELECT @@global.socket` INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; CREATE USER spider@localhost IDENTIFIED BY 'pwd' ; GRANT ALL ON test.* TO spider@localhost; eval CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET "$SOCKET" , DATABASE 'test' , USER 'spider' , PASSWORD 'pwd' ); SET spider_same_server_link=1; CREATE TABLE t1 (c INT , KEY (c)) ENGINE=Spider PARTITION BY KEY (c) PARTITIONS 1; CREATE TABLE t2 (c INT ,d INT ) ENGINE=Spider COMMENT= 'WRAPPER "mysql",SRV "srv",TABLE "t1"' ; LOCK TABLES t2 READ ,t1 WRITE; And for second testcase (hangs always): --let $SOCKET= `SELECT @@global.socket` INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; CREATE USER spider@localhost IDENTIFIED BY 'pwd' ; GRANT ALL ON test.* TO spider@localhost; eval CREATE SERVER srv FOREIGN DATA WRAPPER MYSQL OPTIONS (SOCKET "$SOCKET" , DATABASE 'test' , USER 'spider' , PASSWORD 'pwd' ); SET spider_same_server_link=1; CREATE TABLE t1 (c INT , KEY (c)) ENGINE=Spider; CREATE TABLE t2 (c INT ,d INT ) ENGINE=Spider COMMENT= 'WRAPPER "mysql",SRV "srv",TABLE "t1"' ; LOCK TABLES t2 READ ,t1 WRITE;
            ycp Yuchen Pei added a comment - - edited

            The assertion failure seems to have the same cause as MDEV-33031.
            The hanging seems to be a duplicate of as MDEV-32451.

            Therefore we aim to fix the assertion failure in this issue, and leave the hanging to MDEV-32451.

            A poc fix of the assertion failure could be:

            modified   storage/spider/spd_trx.cc
            @@ -1150,15 +1150,22 @@ SPIDER_TRX *spider_get_trx(
               int *error_num
             ) {
               int roop_count = 0, roop_count2;
            -  SPIDER_TRX *trx;
            +  SPIDER_TRX *trx= NULL;
               SPIDER_SHARE *tmp_share;
               SPIDER_WIDE_HANDLER *tmp_wide_handler;
               pthread_mutex_t *udf_table_mutexes;
               DBUG_ENTER("spider_get_trx");
             
            +  if (thd)
            +  {
            +    if (thd != current_thd)
            +      mysql_mutex_lock(&thd->LOCK_thd_data);
            +    trx = (SPIDER_TRX*) thd_get_ha_data(thd, spider_hton_ptr);
            +    if (thd != current_thd)
            +      mysql_mutex_unlock(&thd->LOCK_thd_data);
            +  }
               if (
            -    !thd ||
            -    !(trx = (SPIDER_TRX*) thd_get_ha_data(thd, spider_hton_ptr))
            +    !trx
               ) {
                 DBUG_PRINT("info",("spider create new trx"));
                 if (!(trx = (SPIDER_TRX *)
            

            ycp Yuchen Pei added a comment - - edited The assertion failure seems to have the same cause as MDEV-33031 . The hanging seems to be a duplicate of as MDEV-32451 . Therefore we aim to fix the assertion failure in this issue, and leave the hanging to MDEV-32451 . A poc fix of the assertion failure could be: modified storage/spider/spd_trx.cc @@ -1150,15 +1150,22 @@ SPIDER_TRX *spider_get_trx( int *error_num ) { int roop_count = 0, roop_count2; - SPIDER_TRX *trx; + SPIDER_TRX *trx= NULL; SPIDER_SHARE *tmp_share; SPIDER_WIDE_HANDLER *tmp_wide_handler; pthread_mutex_t *udf_table_mutexes; DBUG_ENTER("spider_get_trx"); + if (thd) + { + if (thd != current_thd) + mysql_mutex_lock(&thd->LOCK_thd_data); + trx = (SPIDER_TRX*) thd_get_ha_data(thd, spider_hton_ptr); + if (thd != current_thd) + mysql_mutex_unlock(&thd->LOCK_thd_data); + } if ( - !thd || - !(trx = (SPIDER_TRX*) thd_get_ha_data(thd, spider_hton_ptr)) + !trx ) { DBUG_PRINT("info",("spider create new trx")); if (!(trx = (SPIDER_TRX *)
            ycp Yuchen Pei added a comment -

            I confirm that:

            • the assertion failure disappears when applying the MDEV-34636 patch, indicating this issue is duplicated by MDEV-34636
            • set global lock_wait_timeout=1; turns the hang into timeout exceeded, indicating this issue duplicates MDEV-32451

            Therefore I close this ticket as a duplication.

            ycp Yuchen Pei added a comment - I confirm that: the assertion failure disappears when applying the MDEV-34636 patch, indicating this issue is duplicated by MDEV-34636 set global lock_wait_timeout=1; turns the hang into timeout exceeded, indicating this issue duplicates MDEV-32451 Therefore I close this ticket as a duplication.

            People

              ycp Yuchen Pei
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.