[MDEV-24097] galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed - Jira

Stepan Patryshev (Inactive) created issue - 2020-11-02 17:22

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:22

Field	Original Value	New Value
Link		This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:22

Link

This issue blocks MENT-565 [ MENT-565 ]

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:24

Description

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

[BB, 10.2 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784], [BB, 10.6 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles123-amd64/builds/5748/steps/mtr-galera/logs/stdio]: "Failed to start mysqld.3".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784/steps/mtr-galera/logs/stdio]*:
{code:title=10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart - using different config file

Test ended at 2020-10-22 11:23:56

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap

worker[1] > Restart - using different config file

Failed to start mysqld.3

worker[1] > Restart - using different config file
- skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap

WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
Warning: Memory not freed: 1427648
{code}

[BB, 10.4 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999] and [BB, 10.5 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-focal-amd64/builds/1527]: "mysql_shutdown failed".
*[stdio.log|^http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999/steps/mtr-galera/logs/stdio]*:
{code:title=10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
Test ended at 2020-10-30 08:09:16

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;
{code}

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

[BB 10.2 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784] and [BB 10.6 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles123-amd64/builds/5748/steps/mtr-galera/logs/stdio]: "Failed to start mysqld.3".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784/steps/mtr-galera/logs/stdio]*:
{code:title=10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart - using different config file

Test ended at 2020-10-22 11:23:56

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap

worker[1] > Restart - using different config file

Failed to start mysqld.3

worker[1] > Restart - using different config file
- skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap

WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
Warning: Memory not freed: 1427648
{code}

[BB 10.4 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999] and [BB 10.5 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-focal-amd64/builds/1527]: "mysql_shutdown failed".
*[stdio.log|^http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999/steps/mtr-galera/logs/stdio]*:
{code:title=10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
Test ended at 2020-10-30 08:09:16

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;
{code}

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:24

Description

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

[BB 10.2 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784] and [BB 10.6 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles123-amd64/builds/5748/steps/mtr-galera/logs/stdio]: "Failed to start mysqld.3".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784/steps/mtr-galera/logs/stdio]*:
{code:title=10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart - using different config file

Test ended at 2020-10-22 11:23:56

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap

worker[1] > Restart - using different config file

Failed to start mysqld.3

worker[1] > Restart - using different config file
- skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap

WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
Warning: Memory not freed: 1427648
{code}

[BB 10.4 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999] and [BB 10.5 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-focal-amd64/builds/1527]: "mysql_shutdown failed".
*[stdio.log|^http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999/steps/mtr-galera/logs/stdio]*:
{code:title=10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
Test ended at 2020-10-30 08:09:16

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;
{code}

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

[BB 10.2 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784] and [BB 10.6 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles123-amd64/builds/5748/steps/mtr-galera/logs/stdio]: "Failed to start mysqld.3".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784/steps/mtr-galera/logs/stdio]*:
{code:title=10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart - using different config file

Test ended at 2020-10-22 11:23:56

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap

worker[1] > Restart - using different config file

Failed to start mysqld.3

worker[1] > Restart - using different config file
- skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap

WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
Warning: Memory not freed: 1427648
{code}

[BB 10.4 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999] and [BB 10.5 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-focal-amd64/builds/1527]: "mysql_shutdown failed".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999/steps/mtr-galera/logs/stdio]*:
{code:title=10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
Test ended at 2020-10-30 08:09:16

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;
{code}

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:33

Attachment		201023_10_4_ES_stdout.log [ 54603 ]
Attachment		201023_10_4_ES_galera_safe_to_bootstrap.zip [ 54604 ]

Stepan Patryshev (Inactive) made changes - 2020-11-02 17:34

Assignee

Stepan Patryshev [ stepan.patryshev ]

Jan Lindström [ jplindst ]

Roel Van de Paar made changes - 2020-11-10 10:25

Fix Version/s

10.6 [ 24028 ]

Jan Lindström (Inactive) made changes - 2020-11-17 06:17

Assignee

Jan Lindström [ jplindst ]

Seppo Jaakola [ seppo ]

Seppo Jaakola made changes - 2020-11-18 15:13

Status

Open [ 1 ]

In Progress [ 3 ]

Jan Lindström (Inactive) made changes - 2020-11-19 10:57

Assignee

Seppo Jaakola [ seppo ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2020-11-23 05:48

issue.field.resolutiondate

2020-11-23 05:48:43.0

2020-11-23 05:48:43.829

Jan Lindström (Inactive) made changes - 2020-11-23 05:48

Fix Version/s		10.2.37 [ 25112 ]
Fix Version/s		10.3.28 [ 25111 ]
Fix Version/s		10.4.18 [ 25110 ]
Fix Version/s		10.5.9 [ 25109 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Stepan Patryshev (Inactive) made changes - 2020-11-23 07:21

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Stepan Patryshev (Inactive) made changes - 2020-11-23 07:21

Link

This issue blocks MENT-565 [ MENT-565 ]

Stepan Patryshev (Inactive) made changes - 2020-12-22 15:34

Attachment

MDEV-24097_104_logs_201215.zip [ 55431 ]

Stepan Patryshev (Inactive) added a comment - 2020-12-22 15:34 - edited

Reopened since it was sporadaically reproduced on BB 10.4 CS, BB 10.5 CS and BB 10.6 CS: "mysql_shutdown failed".
stdio.log:

10.4.18, d4c35fb21b11f05b317d2e98de42f9a2cab46c27, kvm-rpm-rhel8-amd64
galera_3nodes.galera_safe_to_bootstrap 'innodb' w2 [ fail ]
Test ended at 2020-12-15 06:43:24

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;

- saving '/dev/shm/var/2/log/galera_3nodes.galera_safe_to_bootstrap-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

worker[2] > Restart - not started
worker[2] > Restart - not started
worker[2] > Restart - not started

Server logs.

Stepan Patryshev (Inactive) added a comment - 2020-12-22 15:34 - edited Reopened since it was sporadaically reproduced on BB 10.4 CS , BB 10.5 CS and BB 10.6 CS : "mysql_shutdown failed". stdio.log : 10.4.18, d4c35fb21b11f05b317d2e98de42f9a2cab46c27, kvm-rpm-rhel8-amd64 galera_3nodes.galera_safe_to_bootstrap 'innodb' w2 [ fail ] Test ended at 2020-12-15 06:43:24 CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap mysqltest: In included file "./include/shutdown_mysqld.inc": included from /usr/share/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50: At line 40: mysql_shutdown failed The result from queries just before the failure was: connection node_2; connection node_1; connection node_1; connection node_2; connection node_3; connection node_1; CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB; INSERT INTO t1 VALUES (1); include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0'] connection node_2; - saving '/dev/shm/var/2/log/galera_3nodes.galera_safe_to_bootstrap-innodb/' to '/dev/shm/var/log/galera_3nodes.galera_safe_to_bootstrap-innodb/' Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)... worker[2] > Restart - not started worker[2] > Restart - not started worker[2] > Restart - not started Server logs .

Stepan Patryshev (Inactive) made changes - 2020-12-22 15:34

Resolution	Fixed [ 1 ]
Status	Closed [ 6 ]	Stalled [ 10000 ]

Stepan Patryshev (Inactive) made changes - 2020-12-22 15:35

Fix Version/s		10.2 [ 14601 ]
Fix Version/s		10.3 [ 22126 ]
Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]
Fix Version/s	10.5.9 [ 25109 ]
Fix Version/s	10.4.18 [ 25110 ]
Fix Version/s	10.3.28 [ 25111 ]
Fix Version/s	10.2.37 [ 25112 ]

Stepan Patryshev (Inactive) made changes - 2020-12-22 15:37

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Stepan Patryshev (Inactive) made changes - 2020-12-23 17:28

Link

This issue relates to ~~MDEV-24481~~ [ ~~MDEV-24481~~ ]

Julien Fritsch made changes - 2021-05-05 12:59

Reporter

Stepan Patryshev [ stepan.patryshev ]

Ramesh Sivaraman [ JIRAUSER48189 ]

Julien Fritsch made changes - 2021-05-05 13:01

Priority

Critical [ 2 ]

Major [ 3 ]

Sergei Golubchik made changes - 2021-05-25 12:51

Fix Version/s

10.6 [ 24028 ]

Jan Lindström (Inactive) added a comment - 2021-06-04 09:50

Test does fail at least on buildbot.

Jan Lindström (Inactive) added a comment - 2021-06-04 09:50 Test does fail at least on buildbot.

Julius Goryavsky made changes - 2021-06-09 02:06

Link

This issue relates to ~~MDEV-25880~~ [ ~~MDEV-25880~~ ]

Sergei Golubchik made changes - 2021-12-06 21:35

Workflow

MariaDB v3 [ 115319 ]

MariaDB v4 [ 143668 ]

Jan Lindström (Inactive) made changes - 2021-12-15 12:22

Description

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails with different errors on different versions.

[BB 10.2 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784] and [BB 10.6 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-zyp-sles123-amd64/builds/5748/steps/mtr-galera/logs/stdio]: "Failed to start mysqld.3".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-stretch-amd64/builds/16784/steps/mtr-galera/logs/stdio]*:
{code:title=10.2.35 21ea14db8cc8c5d88ff804650de7caf984d08a98, kvm-deb-stretch-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]worker[1] > Restart - using different config file

Test ended at 2020-10-22 11:23:56

CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_mariabackup_section
CURRENT_TEST: galera_3nodes.galera_ipv6_rsync
CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap

worker[1] > Restart - using different config file

Failed to start mysqld.3

worker[1] > Restart - using different config file
- skipping '/dev/shm/var/1/log/galera_3nodes.galera_safe_to_bootstrap-innodb/'

Retrying test galera_3nodes.galera_safe_to_bootstrap, attempt(2/3)...

***Warnings generated in error logs during shutdown after running tests: galera_3nodes.galera_safe_to_bootstrap

WSREP_SST: [ERROR] rsync daemon port '16011' has been taken (20201022 11:23:48.467)
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300
2020-10-22 11:23:48 140023005419264 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '127.0.0.1:16011' --datadir '/dev/shm/var/1/mysqld.3/data/' --defaults-file '/dev/shm/var/1/my.cnf' --defaults-group-suffix '.3' --parent '30518' --binlog-index 'mysqld-bin.index' --mysqld-args --defaults-group-suffix=.3 --defaults-file=/dev/shm/var/1/my.cnf --log-output=file --innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx --innodb-locks --innodb-metrics --innodb-buffer-pool-stats --innodb-buffer-page --innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields --innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes --innodb-sys-tables --innodb-sys-virtual --core-file --loose-debug-sync-timeout=300: 16 (Device or resource busy)
2020-10-22 11:23:48 140023253743360 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
2020-10-22 11:23:48 140023253743360 [ERROR] Aborting
Warning: Memory not freed: 1427648
{code}

[BB 10.4 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999] and [BB 10.5 CS|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-focal-amd64/builds/1527]: "mysql_shutdown failed".
*[stdio.log|http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-amd64/builds/10999/steps/mtr-galera/logs/stdio]*:
{code:title=10.4.16 199863d72b7cccaa4c75641c50c45a83b568ab8c, kvm-deb-jessie-amd64}
galera_3nodes.galera_safe_to_bootstrap 'innodb' w1 [ fail ]
Test ended at 2020-10-30 08:09:16

CURRENT_TEST: galera_3nodes.galera_safe_to_bootstrap
mysqltest: In included file "./include/shutdown_mysqld.inc":
included from /usr/share/mysql/mysql-test/suite/galera_3nodes/t/galera_safe_to_bootstrap.test at line 50:
At line 40: mysql_shutdown failed

The result from queries just before the failure was:
connection node_2;
connection node_1;
connection node_1;
connection node_2;
connection node_3;
connection node_1;
CREATE TABLE t1 (f1 INTEGER NOT NULL PRIMARY KEY) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
include/assert_grep.inc [grastate.dat does not have 'safe_to_bootstrap: 0']
connection node_2;
{code}

galera_3nodes suite is unstable in following test cases:

* galera_pc_bootstrap
* galera_ipv6_mariabackup
* galera_ipv6_mariabackup_section
* galera_ipv6_rsync
* galera_ipv6_rsync_section
* galera_ssl_reload
* galera_toi_vote
* galera_wsrep_schema_init

because MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

Jan Lindström (Inactive) made changes - 2021-12-15 12:22

Assignee

Jan Lindström [ jplindst ]

Julius Goryavsky [ sysprg ]

Jan Lindström (Inactive) made changes - 2021-12-15 12:22

Priority

Major [ 3 ]

Critical [ 2 ]

Jan Lindström (Inactive) made changes - 2021-12-15 12:23

Affects Version/s		10.4 [ 22408 ]
Affects Version/s	10.6.0 [ 24431 ]
Affects Version/s	10.5.7 [ 25019 ]
Affects Version/s	10.4.16 [ 25020 ]
Affects Version/s	10.2.35 [ 25022 ]

Jan Lindström (Inactive) made changes - 2021-12-15 12:23

Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]

Jan Lindström (Inactive) made changes - 2021-12-15 12:25

Summary

galera_3nodes.galera_safe_to_bootstrap MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

Jan Lindström (Inactive) made changes - 2021-12-15 12:26

Description

galera_3nodes suite is unstable in following test cases:

* galera_pc_bootstrap
* galera_ipv6_mariabackup
* galera_ipv6_mariabackup_section
* galera_ipv6_rsync
* galera_ipv6_rsync_section
* galera_ssl_reload
* galera_toi_vote
* galera_wsrep_schema_init

because MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

galera_3nodes suite is unstable in following test cases:

* galera_pc_bootstrap
* galera_ipv6_mariabackup
* galera_ipv6_mariabackup_section
* galera_ipv6_rsync
* galera_ipv6_rsync_section
* galera_ssl_reload
* galera_toi_vote
* galera_wsrep_schema_init

because MTR sporadaically fails: Failed to start mysqld or mysql_shutdown failed

Latest occurrence in 10.6 as a example: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-bionic-amd64/builds/8868/steps/mtr-galera/logs/stdio

Julius Goryavsky made changes - 2021-12-16 10:48

Fix Version/s		10.2 [ 14601 ]
Fix Version/s		10.3 [ 22126 ]
Fix Version/s		10.4 [ 22408 ]
Fix Version/s		10.5 [ 23123 ]
Fix Version/s		10.6 [ 24028 ]
Fix Version/s		10.7 [ 24805 ]

Julius Goryavsky made changes - 2021-12-16 10:48

Affects Version/s		10.7.1 [ 26120 ]
Affects Version/s		10.6.5 [ 26034 ]
Affects Version/s		10.5.13 [ 26026 ]
Affects Version/s		10.4.22 [ 26031 ]
Affects Version/s		10.3.32 [ 26029 ]
Affects Version/s		10.2.41 [ 26032 ]
Affects Version/s	10.4 [ 22408 ]

Julius Goryavsky made changes - 2021-12-16 10:48

Status

Stalled [ 10000 ]

In Progress [ 3 ]

Julius Goryavsky made changes - 2021-12-21 01:18

Assignee	Julius Goryavsky [ sysprg ]	Jan Lindström [ jplindst ]
Status	In Progress [ 3 ]	In Review [ 10002 ]

Jan Lindström (Inactive) added a comment - 2021-12-21 08:30

I'm ok with these changes. Note that change on mysql-test/suite/galera/t/galera_autoinc_sst_xtrabackup.cnf has no effect (please check is this intentional).

Jan Lindström (Inactive) added a comment - 2021-12-21 08:30 I'm ok with these changes. Note that change on mysql-test/suite/galera/t/galera_autoinc_sst_xtrabackup.cnf has no effect (please check is this intentional).

Jan Lindström (Inactive) made changes - 2021-12-21 08:30

Assignee	Jan Lindström [ jplindst ]	Ramesh Sivaraman [ JIRAUSER48189 ]
Status	In Review [ 10002 ]	Stalled [ 10000 ]

Jan Lindström (Inactive) made changes - 2021-12-21 08:30

Status

Stalled [ 10000 ]

In Testing [ 10301 ]

Julius Goryavsky made changes - 2022-01-11 11:16

Assignee

Ramesh Sivaraman [ JIRAUSER48189 ]

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2022-01-17 03:05

Link

This issue relates to ~~MDEV-27524~~ [ ~~MDEV-27524~~ ]

Julius Goryavsky made changes - 2022-01-17 03:10

Link

This issue split to ~~MDEV-27524~~ [ ~~MDEV-27524~~ ]

Julius Goryavsky made changes - 2022-01-17 03:10

Link

This issue relates to ~~MDEV-27524~~ [ ~~MDEV-27524~~ ]

Julius Goryavsky added a comment - 2022-01-28 00:06

Fixed and merged, remaining binlogs issue moved to a separate task (~~MDEV-27524~~)

Julius Goryavsky added a comment - 2022-01-28 00:06 Fixed and merged, remaining binlogs issue moved to a separate task ( MDEV-27524 )

Julius Goryavsky made changes - 2022-01-28 00:06

issue.field.resolutiondate

2022-01-28 00:06:55.0

2022-01-28 00:06:55.834

Julius Goryavsky made changes - 2022-01-28 00:07

Fix Version/s		10.2.42 [ 26803 ]
Fix Version/s		10.3.33 [ 26805 ]
Fix Version/s		10.4.23 [ 26807 ]
Fix Version/s		10.5.14 [ 26809 ]
Fix Version/s		10.6.6 [ 26811 ]
Fix Version/s		10.7.2 [ 26813 ]
Fix Version/s		10.8.1 [ 26815 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Fix Version/s	10.6 [ 24028 ]
Fix Version/s	10.7 [ 24805 ]
Resolution		Fixed [ 1 ]
Status	In Testing [ 10301 ]	Closed [ 6 ]

Julien Fritsch made changes - 2022-01-31 08:24

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Julius Goryavsky made changes - 2022-02-18 13:46

Link

This issue relates to ~~MDEV-23747~~ [ ~~MDEV-23747~~ ]

MariaDB Server

galera_3nodes suite tests in MTR sporadically fails: Failed to start mysqld or mysql_shutdown failed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration