Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23576

galera_3nodes.galera_ipv6_mysqldump MTR failed: WSREP_SST: [ERROR] rsync daemon port '16008' has been taken

Details

    Description

      galera_3nodes.galera_ipv6_mysqldump failed on bb 10.2 and bb 10.3: "WSREP_SST: [ERROR] rsync daemon port '16008' has been taken".

      stdio.log:

      10.2.34 0be70a1b773ce66ef1803fcce19522fd9c60c07d

      galera_3nodes.galera_ipv6_mysqldump 'innodb' w1 [ fail ]
              Test ended at 2020-08-25 13:32:03
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
      CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup
      CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
       
       
      Failed to start mysqld.2
       
       
       - skipping '/dev/shm/var/1/log/galera_3nodes.galera_ipv6_mysqldump-innodb/'
       
      Retrying test galera_3nodes.galera_ipv6_mysqldump, attempt(2/3)...
       
      ***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_mysqldump
       
      WSREP_SST: [ERROR] rsync daemon port '16008' has been taken (20200825 13:31:53.933)
      2020-08-25 13:31:53 140068462999296 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16008' --datadir '/dev/shm/var/1/mysqld.2/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.2' --parent '27002' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300
      2020-08-25 13:31:53 140068462999296 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16008' --datadir '/dev/shm/var/1/mysqld.2/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.2' --parent '27002' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.2 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --bind-address=:: --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
      2020-08-25 13:31:53 140068463302400 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
      2020-08-25 13:31:53 140068463302400 [ERROR] Aborting
      Warning: Memory not freed: 1427624
      

      On bb 10.4 and bb 10.5 it failed with another output: "mysql_shutdown failed".

      stdio.log:

      10.4.15 88e70f4caea34e0d7677b1fa646151b8b87dd3ab

      galera_3nodes.galera_ipv6_mysqldump 'innodb' w1 [ fail ]
              Test ended at 2020-08-25 06:58:54
       
      CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
      mysqltest: In included file "./include/shutdown_mysqld.inc": 
      included from ./include/restart_mysqld.inc at line 10:
      included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 83:
      At line 40: mysql_shutdown failed
       
      The result from queries just before the failure was:
      < snip >
      connection node_1;
      CREATE USER 'sst';
      GRANT ALL PRIVILEGES ON *.* TO 'sst';
      SET GLOBAL wsrep_sst_auth = 'sst:';
      connection node_2;
      SET GLOBAL wsrep_sst_method = 'mysqldump';
      Shutting down server ...
      connection node_1;
      Cleaning var directory ...
      CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB;
      INSERT INTO t1 VALUES (1);
      connection node_2;
      Starting server ...
      SELECT COUNT(*) = 1 FROM t1;
      COUNT(*) = 1
      1
      DROP TABLE t1;
      SELECT VARIABLE_VALUE LIKE '%[::1]%' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_incoming_addresses';
      VARIABLE_VALUE LIKE '%[::1]%'
      

      Attachments

        Issue Links

          Activity

            sysprg Julius Goryavsky added a comment - This is a duplicate of MDEV-21770 , corrected here: https://github.com/MariaDB/server/commit/ac3c19de6967abbaaa661f72526b233d11bd6868

            Because the test keeps failing even after the supposed MDEV-21770 fix, I disabled the test on 10.4.

            marko Marko Mäkelä added a comment - Because the test keeps failing even after the supposed MDEV-21770 fix, I disabled the test on 10.4.
            sysprg Julius Goryavsky added a comment - - edited

            marko Thanks, this is something new, not related to MDEV-21770 and original MDEV-23576 - in this case it crashes during server shutdown. I'll figure out why this is happening - this is some kind of new problem, different from the one in MDEV-21770 and from the original MDEV-23576:

            galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ fail ]
                    Test ended at 2020-10-22 06:12:07
             
            CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump
            mysqltest: In included file "./include/shutdown_mysqld.inc": 
            included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 44:
            At line 40: mysql_shutdown failed
            

            and (node-2 log):

            2020-10-22  6:12:06 2 [Note] WSREP: Non-primary view
            2020-10-22  6:12:06 2 [Note] WSREP: Server status change disconnecting -> disconnected
            2020-10-22  6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2020-10-22  6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
            2020-10-22  6:12:06 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2
            2020-10-22  6:12:06 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
            2020-10-22  6:12:06 0 [Note] WSREP: killing local connection: 15
            

            sysprg Julius Goryavsky added a comment - - edited marko Thanks, this is something new, not related to MDEV-21770 and original MDEV-23576 - in this case it crashes during server shutdown. I'll figure out why this is happening - this is some kind of new problem, different from the one in MDEV-21770 and from the original MDEV-23576 : galera_3nodes.galera_ipv6_mysqldump 'innodb' w2 [ fail ] Test ended at 2020-10-22 06:12:07   CURRENT_TEST: galera_3nodes.galera_ipv6_mysqldump mysqltest: In included file "./include/shutdown_mysqld.inc": included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_ipv6_mysqldump.test at line 44: At line 40: mysql_shutdown failed and (node-2 log): 2020-10-22 6:12:06 2 [Note] WSREP: Non-primary view 2020-10-22 6:12:06 2 [Note] WSREP: Server status change disconnecting -> disconnected 2020-10-22 6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-10-22 6:12:06 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-10-22 6:12:06 2 [Note] WSREP: Applier thread exiting ret: 0 thd: 2 2020-10-22 6:12:06 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication) 2020-10-22 6:12:06 0 [Note] WSREP: killing local connection: 15

            Closed as duplicate of the MDEV-21770, but created a separate task for the detected problem during shutdown: https://jira.mariadb.org/browse/MDEV-24036

            sysprg Julius Goryavsky added a comment - Closed as duplicate of the MDEV-21770 , but created a separate task for the detected problem during shutdown: https://jira.mariadb.org/browse/MDEV-24036

            People

              sysprg Julius Goryavsky
              stepan.patryshev Stepan Patryshev (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.