Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24097

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Stalled (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.2.35, 10.4.16, 10.5.7, 10.6.0
    • Fix Version/s: 10.2, 10.3, 10.4, 10.5, 10.6
    • Component/s: Galera, Tests
    • Labels:
      None
    • Environment:
      kvm-deb-stretch-amd64 kvm-deb-jessie-amd64

      Description

      galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

      BB 10.2 CS and BB 10.6 CS: "Failed to start mysqld.3".
      stdio.log:

      10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64

      galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart  - using different config file
       
              Test ended at 2020-10-22 11:23:56
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
      CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
      CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
       
      worker[1] > Restart  - using different config file
       
      Failed to start mysqld.3
       
       
      worker[1] > Restart  - using different config file
       - skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'
       
      Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...
       
      ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap
       
      WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
      2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
      2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
      2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
      2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
      Warning: Memory not freed: 1427648
      

      BB 10.4 CS and BB 10.5 CS: "mysql_shutdown failed".
      stdio.log:

      10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64

      galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
              Test ended at 2020-10-30 08:09:16
       
      CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
      mysqltest: In included file "./include/shutdown_mysqld.inc": 
      included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
      At line 40: mysql_shutdown failed
       
      The result from queries just before the failure was:
      connection node_2;
      connection node_1;
      connection node_1;
      connection node_2;
      connection node_3;
      connection node_1;
      CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (1);
      include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
      include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
      include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
      connection node_2;
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              stepan.patryshev Stepan Patryshev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: