Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24097

galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

Details

    Description

      galera_3nodes suite is unstable in following test cases:

      • galera_pc_bootstrap
      • galera_ipv6_mariabackup
      • galera_ipv6_mariabackup_section
      • galera_ipv6_rsync
      • galera_ipv6_rsync_section
      • galera_ssl_reload
      • galera_toi_vote
      • galera_wsrep_schema_init

      because MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

      Latest occurrence in 10.6 as a example: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/builds/8868/steps/mtr-galera/logs/stdio

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23747 socket.ssl_ca argument being ignored.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25880 rsync may be mistakenly killed when overlapping SST

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24481 galera_3nodes.galera_vote_rejoin_mysqldump MTR failed: mysql_shutdown failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          split to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27524 Incorrect binlogs after Galera SST using rsync and mariabackup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited

            Reopened since it was sporadaically reproduced on BB 10.4 CS, BB 10.5 CS and BB 10.6 CS: "mysql_shutdown failed".
            stdio.log:

            10.4.18, d4c35fb21b11f05b317d2e98de42f9a2cab46c27, kvm-rpm-rhel8-amd64

            		 
            galera_3nodes.galera_safe_to_bootstrap 'innodb' w2 [ fail ]
            		 
                    Test ended at 2020-12-15 06:43:24
            		 
             
            		 
            CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
            		 
            mysqltest: In included file "./include/shutdown_mysqld.inc": 
            included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
            		 
            At line 40: mysql_shutdown failed
            		 
             
            		 
            The result from queries just before the failure was:
            		 
            connection node_2;
            		 
            connection node_1;
            		 
            connection node_1;
            		 
            connection node_2;
            		 
            connection node_3;
            		 
            connection node_1;
            		 
            CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
            		 
            INSERT INTO t1 VALUES (1);
            		 
            include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
            		 
            include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
            		 
            include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
            		 
            connection node_2;
            		 
             
            		 
             - saving '/dev/shm/var/2/log/galera_3nodes.galera_safe_to_bootstrap-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'
            		 
             
            		 
            Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...
            		 
             
            		 
            worker[2] > Restart  - not started
            		 
            worker[2] > Restart  - not started
            		 
            worker[2] > Restart  - not started

            Server logs.

            stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopened since it was sporadaically reproduced on BB 10.4 CS , BB 10.5 CS and BB 10.6 CS : "mysql_shutdown failed". stdio.log : 10.4.18, d4c35fb21b11f05b317d2e98de42f9a2cab46c27, kvm-rpm-rhel8-amd64 galera_3nodes.galera_safe_to_bootstrap 'innodb' w2 [ fail ] Test ended at 2020-12-15 06:43:24   CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap mysqltest: In included file "./include/shutdown_mysqld.inc": included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50: At line 40: mysql_shutdown failed   The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; connection node_2; connection node_3; connection node_1; CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1); include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] connection node_2;   - saving '/dev/shm/var/2/log/galera_3nodes.galera_safe_to_bootstrap-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'   Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...   worker[2] > Restart - not started worker[2] > Restart - not started worker[2] > Restart - not started Server logs .
            jplindst Jan Lindström (Inactive) added a comment -

            Test does fail at least on buildbot.

            jplindst Jan Lindström (Inactive) added a comment - Test does fail at least on buildbot.
            jplindst Jan Lindström (Inactive) added a comment -

            I'm ok with these changes. Note that change on mysql-test/suite/galera/t/galera_autoinc_sst_xtrabackup.cnf has no effect (please check is this intentional).

            jplindst Jan Lindström (Inactive) added a comment - I'm ok with these changes. Note that change on mysql-test/suite/galera/t/galera_autoinc_sst_xtrabackup.cnf has no effect (please check is this intentional).
            sysprg Julius Goryavsky added a comment -

            Fixed and merged, remaining binlogs issue moved to a separate task (MDEV-27524)

            sysprg Julius Goryavsky added a comment - Fixed and merged, remaining binlogs issue moved to a separate task ( MDEV-27524 )

            People

              sysprg Julius Goryavsky
              ramesh Ramesh Sivaraman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.