Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30232

rpl.rpl_gtid_crash fails sporadically in BB with Timeout wait for SQL thread to catch up with IO thread

Details

    Description

      https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/34174

      10.8 e8a2a70cf

      rpl.rpl_gtid_crash 'innodb,row'          w4 [ fail ]
              Test ended at 2022-03-08 06:14:34
       
      CURRENT_TEST: rpl.rpl_gtid_crash
      mysqltest: At line 367: Timeout wait for SQL thread to catch up with IO thread
       
      The result from queries just before the failure was:
      < snip >
      call mtr.add_suppression("Unexpected change of master binlog file name in the middle of GTID");
      set sql_log_bin= 1;
      connection server_1;
      SET GLOBAL debug_dbug="+d,inject_error_writing_xid";
      BEGIN;
      INSERT INTO t1 VALUES (11);
      COMMIT;
      ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
      SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
      COMMIT;
      Got one of the listed errors
      SELECT @@GLOBAL.server_id;
      @@GLOBAL.server_id
      3
      SELECT * from t1 WHERE a > 10 ORDER BY a;
      a
      gtid_check
      Binlog pos ok
      # Wait 30 seconds for SQL thread to catch up with IO thread
      connection server_2;
      

      Also seen to have failed at the second wait for the SQL thread:

      10.6 c4ce012e4

      rpl.rpl_gtid_crash 'innodb,row'          w4 [ fail ]
              Test ended at 2022-11-07 19:05:14
       
      CURRENT_TEST: rpl.rpl_gtid_crash
      mysqltest: At line 464: Timeout wait for SQL thread to catch up with IO thread
       
      The result from queries just before the failure was:
      < snip >
      BEGIN;
      INSERT INTO t1 VALUES (21);
      COMMIT;
      ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
      SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
      COMMIT;
      Got one of the listed errors
      SELECT @@GLOBAL.server_id;
      @@GLOBAL.server_id
      1
      SELECT * from t1 WHERE a > 10 ORDER BY a;
      a
      13
      14
      gtid_check
      Binlog pos ok
      gtid_check
      Current pos ok
      # Wait 30 seconds for SQL thread to catch up with IO thread
      connection server_2;
      

      Attachments

        Issue Links

          Activity

            angelique.sklavounos Angelique Sklavounos (Inactive) created issue -
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            Field Original Value New Value
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            Description {noformat:title=10.8 e8a2a70cf}
            rpl.rpl_gtid_crash 'innodb,row' w4 [ fail ]
                    Test ended at 2022-03-08 06:14:34

            CURRENT_TEST: rpl.rpl_gtid_crash
            mysqltest: At line 367: Timeout wait for SQL thread to catch up with IO thread

            The result from queries just before the failure was:
            < snip >
            call mtr.add_suppression("Unexpected change of master binlog file name in the middle of GTID");
            set sql_log_bin= 1;
            connection server_1;
            SET GLOBAL debug_dbug="+d,inject_error_writing_xid";
            BEGIN;
            INSERT INTO t1 VALUES (11);
            COMMIT;
            ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
            SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
            COMMIT;
            Got one of the listed errors
            SELECT @@GLOBAL.server_id;
            @@GLOBAL.server_id
            3
            SELECT * from t1 WHERE a > 10 ORDER BY a;
            a
            gtid_check
            Binlog pos ok
            # Wait 30 seconds for SQL thread to catch up with IO thread
            connection server_2;
            {noformat}

            Also seen to have failed at the second wait for the SQL thread:
            {noformat}
            rpl.rpl_gtid_crash 'innodb,row' w4 [ fail ]
                    Test ended at 2022-11-07 19:05:14

            CURRENT_TEST: rpl.rpl_gtid_crash
            mysqltest: At line 464: Timeout wait for SQL thread to catch up with IO thread

            The result from queries just before the failure was:
            < snip >
            BEGIN;
            INSERT INTO t1 VALUES (21);
            COMMIT;
            ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
            SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
            COMMIT;
            Got one of the listed errors
            SELECT @@GLOBAL.server_id;
            @@GLOBAL.server_id
            1
            SELECT * from t1 WHERE a > 10 ORDER BY a;
            a
            13
            14
            gtid_check
            Binlog pos ok
            gtid_check
            Current pos ok
            # Wait 30 seconds for SQL thread to catch up with IO thread
            connection server_2;
            {noformat}
            https://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2/builds/34174
            {noformat:title=10.8 e8a2a70cf}
            rpl.rpl_gtid_crash 'innodb,row' w4 [ fail ]
                    Test ended at 2022-03-08 06:14:34

            CURRENT_TEST: rpl.rpl_gtid_crash
            mysqltest: At line 367: Timeout wait for SQL thread to catch up with IO thread

            The result from queries just before the failure was:
            < snip >
            call mtr.add_suppression("Unexpected change of master binlog file name in the middle of GTID");
            set sql_log_bin= 1;
            connection server_1;
            SET GLOBAL debug_dbug="+d,inject_error_writing_xid";
            BEGIN;
            INSERT INTO t1 VALUES (11);
            COMMIT;
            ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
            SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
            COMMIT;
            Got one of the listed errors
            SELECT @@GLOBAL.server_id;
            @@GLOBAL.server_id
            3
            SELECT * from t1 WHERE a > 10 ORDER BY a;
            a
            gtid_check
            Binlog pos ok
            # Wait 30 seconds for SQL thread to catch up with IO thread
            connection server_2;
            {noformat}

            Also seen to have failed at the second wait for the SQL thread:
            {noformat:title=10.6 c4ce012e4}
            rpl.rpl_gtid_crash 'innodb,row' w4 [ fail ]
                    Test ended at 2022-11-07 19:05:14

            CURRENT_TEST: rpl.rpl_gtid_crash
            mysqltest: At line 464: Timeout wait for SQL thread to catch up with IO thread

            The result from queries just before the failure was:
            < snip >
            BEGIN;
            INSERT INTO t1 VALUES (21);
            COMMIT;
            ERROR HY000: Error writing file 'master-bin' (errno: 28 "No space left on device")
            SET GLOBAL debug_dbug="+d,crash_dispatch_command_before";
            COMMIT;
            Got one of the listed errors
            SELECT @@GLOBAL.server_id;
            @@GLOBAL.server_id
            1
            SELECT * from t1 WHERE a > 10 ORDER BY a;
            a
            13
            14
            gtid_check
            Binlog pos ok
            gtid_check
            Current pos ok
            # Wait 30 seconds for SQL thread to catch up with IO thread
            connection server_2;
            {noformat}
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.8 [ 26121 ]
            Fix Version/s 10.9 [ 26905 ]
            Fix Version/s 10.10 [ 27530 ]
            Fix Version/s 10.11 [ 27614 ]
            Fix Version/s 11.0 [ 28320 ]
            Affects Version/s 10.4 [ 22408 ]
            Affects Version/s 10.5 [ 23123 ]
            Affects Version/s 10.6 [ 24028 ]
            Affects Version/s 10.8 [ 26121 ]
            Affects Version/s 10.9 [ 26905 ]
            Affects Version/s 10.10 [ 27530 ]
            Affects Version/s 10.11 [ 27614 ]
            Affects Version/s 11.0 [ 28320 ]
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.8 [ 26121 ]
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            angelique.sklavounos Angelique Sklavounos (Inactive) made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.9 [ 26905 ]
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 10.10 [ 27530 ]
            knielsen Kristian Nielsen made changes -
            Assignee Angelique Sklavounos [ JIRAUSER50741 ] Kristian Nielsen [ knielsen ]
            knielsen Kristian Nielsen made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            knielsen Kristian Nielsen made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            knielsen Kristian Nielsen made changes -
            Fix Version/s 10.5.25 [ 29626 ]
            Fix Version/s 10.6.18 [ 29627 ]
            Fix Version/s 10.11.8 [ 29630 ]
            Fix Version/s 11.5.1 [ 29634 ]
            Fix Version/s 11.4.1 [ 29523 ]
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.11 [ 27614 ]
            Fix Version/s 11.0 [ 28320 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            JIraAutomate JiraAutomate made changes -
            Fix Version/s 11.0.6 [ 29628 ]
            Fix Version/s 11.1.5 [ 29629 ]
            Fix Version/s 11.2.4 [ 29631 ]
            Fix Version/s 11.4.2 [ 29633 ]
            knielsen Kristian Nielsen made changes -

            People

              knielsen Kristian Nielsen
              angelique.sklavounos Angelique Sklavounos (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.