galera_3nodes.galera_ipv6_rsync_section 'innodb' w2 [ fail ]

        Test ended at 2020-12-15 14:56:12

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

Failed to start mysqld.3

mysqltest failed but provided no output

 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync_section-innodb/'

Retrying test galera_3nodes.galera_ipv6_rsync_section, attempt(2/3)...

worker[2] > Restart  - not started

worker[2] > Restart  - not started

worker[2] > Restart  - not started

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync_section

WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20201215 14:53:09.389)

2020-12-15 14:53:09 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30703' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300

2020-12-15 14:53:09 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '30703' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)

2020-12-15 14:53:09 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

2020-12-15 14:53:09 2 [ERROR] Aborting

worker[2] mysql-test-run: WARNING: Process [mysqld.3 - pid: 387, winpid: 387, exit: 256] died after mysql-test-run waited 11.7 seconds for /dev/shm/var/2/run/mysqld.3.pid to be created.

galera_3nodes.galera_ipv6_rsync_section 'innodb' w2 [ retry-fail ]

        Test ended at 2020-12-15 14:56:28

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

Failed to start mysqld.3

mysqltest failed but provided no output

 - skipping '/dev/shm/var/2/log/galera_3nodes.galera_ipv6_rsync_section-innodb/'

Test galera_3nodes.galera_ipv6_rsync_section has failed 2 times, no more retries!

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync_section

WSREP_SST: [ERROR] rsync daemon port '16031' has been taken (20201215 14:56:14.222)

2020-12-15 14:56:14 0 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '390' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300

2020-12-15 14:56:14 0 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '[::1]:16031' --datadir '/dev/shm/var/2/mysqld.3/data/' --defaults-file '/dev/shm/var/2/my.cnf' --defaults-group-suffix '.3' --parent '390' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/2/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-lock-waits --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)

2020-12-15 14:56:14 2 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.

2020-12-15 14:56:14 2 [ERROR] Aborting

Warning: Memory not freed: 1381120

galera_3nodes.galera_ipv6_rsync_section 'innodb' w1 [ fail ]

        Test ended at 2020-11-23 21:01:56

CURRENT_TEST: galera_3nodes.galera_ipv6_rsync_section

Failed to start mysqld.2

mysqltest failed but provided no output

 - saving '/dev/shm/var/1/log/galera_3nodes.galera_ipv6_rsync_section-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_ipv6_rsync_section-innodb/'

Retrying test galera_3nodes.galera_ipv6_rsync_section, attempt(2/3)...

worker[1] > Restart  - not started

worker[1] > Restart  - not started

worker[1] > Restart  - not started

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_ipv6_rsync_section

2020-11-23 21:01:56 0 [ERROR] Aborting

 1) The ipv6 addresses were incorrectly handled in the SST script

    for rsync (incorrect address substitution for establishing a

    connection, incorrect address substitution for bind, and so on);

 2) Checking the locality of the ip-address in SST scripts did not

    support ipv6 addresses (such as "[::1]"), which were falsely

    identified as non-local ip, which further did not allow running

    two SSTs on different local addresses on the same machine.

    On the other hand, this bug masked some other errors (related

    to handling ipv6 addresses);

 3) The code for checking the locality of the ip address was different

    in the SST scripts for rsync and for mysqldump, with individual

    flaws. This code is now made common and moved to wsrep_sst_common;

 4) Waiting for the start of the transport channel (socat, nc, rsync,

    stunnel) in the wait_for_listen() and check_pid_and_port() functions

    did not process ipv6 addresses correctly in all cases (not for all

    branches);

 5) Waiting for the start of the transport channel (socat, nc, rsync,

    stunnel) in the wait_for_listen() and check_pid_and_port() functions

    for some code branches could give a false positive result due to

    the textual match of prefixes in the port number and/or PID of

    the process;

 6) Waiting for the start of the transport channel (socat, nc, rsync,

    stunnel) was supported through different utilities in SST scripts

    for mariabackup and for rsync, and with various minor flaws in

    the code. Now the code is still different in these scripts, but

    it supports a common set of utilities (lsof, ss, sockstat) and

    is synchronized across patterns that used to check the output

    of  these utilities;

 7) In SST via mariabackup, the signal about readiness to receive data

    is sometimes sent too early - immediately after listen(), and not

    after accept() (which are called by socat or netcat utility).

 8) Checking availability of the some options of some utilities was

    done using the grep pattern, which easily gives false positives;

 9) Common name (CN) for local addresses, if not explicitly specified,

    is now always replaced to "localhost" to avoid the need to generate

    many separate certificates for local addresses of one machine and

    not to depend on which the local address is currently used in test

    (ipv4 or ipv6, etc.);

10) In tests galera_sst_mariabackup_encrypt_with_key_server and

    galera_sst_rsync_encrypt_with_key_server the correct certificate

    is selected to avoid commonname (CN) mismatch problems;

11) Further refactoring to protect against spaces in file names.

12) Further general refactoring to eliminate bash-specific constructs

    or to improve code readability;

13) The code for setting options for the nc (netcat) utility was

    different in different scripts for SST - now it is made identical.

14) Fixed a long time ago broken encryption with xbcrypt combined with

    mariabackup and Added mariabackup support for encryption in encrypt=1

    mode using openssl (instead of xbcrypt utility), which may be enabled

    by the new configuration file option "encypt-format=openssl" placed

    in the [sst] or in the [xtrabackup] section (and enabled by default

    if user prefers [sst] section) - this will allow us to use and to

    test this encryption mode without installing third-party utilities

    from Percona.