Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35064

Thread hang in 'Update' state on 2nd INSERT

Details

    Description

      INSTALL PLUGIN Spider SONAME 'ha_spider.so';
      		 
      CREATE SERVER s FOREIGN DATA WRAPPER mysql OPTIONS(HOST '1');
      		 
      CREATE TABLE t(c INT)ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "s",TABLE "foo"';
      		 
      INSERT INTO t VALUES(1);   # ERROR 1429 (HY000): Unable to connect to foreign data source: s
      		 
      INSERT INTO t VALUES(1);   # Hangs

      Will thread-hang in 'Update' state on the second INSERT.

      CS 11.2.6 e91a79945822def1452787f825e6047c6a64dbd9 (Debug)

      		 
      11.2.6-dbg>SHOW PROCESSLIST;
      		 
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-------------------------+----------+
      		 
      | Id | User        | Host      | db   | Command | Time | State                                              | Info                    | Progress |
      		 
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-------------------------+----------+
      		 
      |  4 | root        | localhost | test | Query   | 2003 | Update                                             | INSERT INTO t VALUES(1) |    0.000 |
      		 
      |  5 | system user |           | NULL | Daemon  | NULL | Spider table background statistics action handler  | NULL                    |    0.000 |
      		 
      |  6 | system user |           | NULL | Daemon  | NULL | Spider table background cardinality action handler | NULL                    |    0.000 |
      		 
      |  9 | root        | localhost | test | Query   |    0 | starting                                           | SHOW PROCESSLIST        |    0.000 |
      		 
      +----+-------------+-----------+------+---------+------+----------------------------------------------------+-------------------------+----------+
      		 
      4 rows in set (0.000 sec)

      Bug confirmed in 10.5 opt, 11.2 dbg and 11.7 opt - deemed to be present in all versions/build types.

      Attachments

        Issue Links

          relates to

          Technical task - A technical task. MDEV-27925 Deprecate spider_connect_retry_count and spider_connect_retry_interval

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29214 ASAN errors in federatedx_txn::txn_commit (or txn_rollback), assertion `server->io_count == 0' failure

          • Major - Major loss of function.
          • Confirmed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29704 Thread hang in 'Waiting for table metadata lock' on CREATE TABLE

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30012 Triple thread hang in states 'NULL', 'NULL' and 'closing tables' on INSERT

          • Major - Major loss of function.
          • Open

          Activity

            Ascending order - Click to sort in descending order
            Roel Roel Van de Paar added a comment -

            MTR Testcase

            INSTALL PLUGIN Spider SONAME 'ha_spider.so';
            		 
            CREATE SERVER s FOREIGN DATA WRAPPER mysql OPTIONS(HOST '1');
            		 
            CREATE TABLE t(c INT)ENGINE=Spider COMMENT='WRAPPER "mysql",SRV "s",TABLE "foo"';
            		 
            --error ER_CONNECT_TO_FOREIGN_DATA_SOURCE
            		 
            INSERT INTO t VALUES(1);
            		 
            INSERT INTO t VALUES(1);  # Hangs

            Roel Roel Van de Paar added a comment - MTR Testcase INSTALL PLUGIN Spider SONAME 'ha_spider.so' ; CREATE SERVER s FOREIGN DATA WRAPPER mysql OPTIONS(HOST '1' ); CREATE TABLE t(c INT )ENGINE=Spider COMMENT= 'WRAPPER "mysql",SRV "s",TABLE "foo"' ; --error ER_CONNECT_TO_FOREIGN_DATA_SOURCE INSERT INTO t VALUES (1); INSERT INTO t VALUES (1); # Hangs
            Roel Roel Van de Paar added a comment -

            MTR Will timeout after 900 seconds and write a core (SIGABRT in _GI__poll). Potentially it may show helpful debugging info.

            Roel Roel Van de Paar added a comment - MTR Will timeout after 900 seconds and write a core ( SIGABRT in _ GI __poll ). Potentially it may show helpful debugging info.
            ycp Yuchen Pei added a comment - - edited

            The cause is that spider retries a failed connection for spider_connect_retry_count times. This variable has a default value of 1000. If we specify the wrong connection info in a different way so that it does not take long to fail, e.g. socket 'abc' instead of host '1', then the hang does not happen, even though we still retry 1000 times.

            I do not think 1000 is a good default for the number of connection retries, and it should be drastically reduced, to something like 3. That will fix this issue.

            The reason that the first insert did not result in a similar hang is because the connection attempt failed earlier, when opening the table, when spider tries to get the data statistics from the remote table. This connection is not retried.

            ycp Yuchen Pei added a comment - - edited The cause is that spider retries a failed connection for spider_connect_retry_count times. This variable has a default value of 1000. If we specify the wrong connection info in a different way so that it does not take long to fail, e.g. socket 'abc' instead of host '1' , then the hang does not happen, even though we still retry 1000 times. I do not think 1000 is a good default for the number of connection retries, and it should be drastically reduced, to something like 3. That will fix this issue. The reason that the first insert did not result in a similar hang is because the connection attempt failed earlier, when opening the table, when spider tries to get the data statistics from the remote table. This connection is not retried.
            ycp Yuchen Pei added a comment -

            Hi holyfoot, ptal thanks

            283ca58829e bb-10.5-mdev-35064 MDEV-35064 Reduce the default spider connect retry counts to 2

            ycp Yuchen Pei added a comment - Hi holyfoot , ptal thanks 283ca58829e bb-10.5-mdev-35064 MDEV-35064 Reduce the default spider connect retry counts to 2
            holyfoot Alexey Botchkov added a comment -

            ok to push.
            Why the retry count was defaulted to 1000 i wonder. If the first connect fails chances that the consequent connect ever succeeds are small.

            holyfoot Alexey Botchkov added a comment - ok to push. Why the retry count was defaulted to 1000 i wonder. If the first connect fails chances that the consequent connect ever succeeds are small.
            ycp Yuchen Pei added a comment -

            thanks for the review - pushed a8cc40d9a42e69cc39242f0a2824cd9cdd852a80 to 10.5

            ycp Yuchen Pei added a comment - thanks for the review - pushed a8cc40d9a42e69cc39242f0a2824cd9cdd852a80 to 10.5

            People

              ycp Yuchen Pei
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.