[MDEV-13549] Galera 3 test failures - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Duplicate
Affects Version/s: 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL)
Fix Version/s: N/A
Component/s: Galera
Labels:
- contribution
- foundation

Description

Collection of known Galera 3 test failures on MariaDB 10.1-10.3.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

galera_gra_log.zip
10 kB
2017-08-16 08:41

Issue Links

is blocked by

MDEV-12234 wsrep.pool_of_threads fails in buildbot on ppc64le

Closed

MDEV-13738 galera_applier_ftwrl_table_alter may fail with '2006: MySQL server has gone away'

Closed

MDEV-13743 galera_toi_truncate may fail with: query 'reap' succeeded - should have failed with errno 1213

Closed

MDEV-13871 galera.galera_unicode_identifiers failed in buildbot with 'Unknown database'

Closed

MDEV-13873 galera.galera_suspend_slave failed in buildbot with wrong error code

Closed

MDEV-13874 galera.galera_wsrep_log_conficts failed in buildbot upon before testcase check

Closed

MDEV-13876 galera.MW-328A failed in buildbot with wrong result or timeout

Closed

MDEV-13877 galera.galera_gcs_fc_limit fails in buildbot with timeout or Interrupted system call

Closed

MDEV-13879 galera.galera_wan fails in buildbot with Stray state UUID msg or with "Transport endpoint is not connected" or with a failure to start a node

Closed

MDEV-13880 galera.galera_var_node_address fails in buildot, Failed to start mysqld or "Transport endpoint is not connected"

Closed

MDEV-13881 galera.partition failed in buildbot with wrong result

Closed

MDEV-13883 galera.query_cache fails in buildbot with wrong result

Closed

MDEV-14142 galera.galera_transaction_read_only fails in buildbot with wrong result

Closed

MDEV-14143 galera.galera_kill_smallchanges, galera.galera_kill_ddl fail in buildbot with "Last Applied Action message in non-primary configuration from member 0"

Closed

MDEV-14145 galera.MW-328B failed in buildbot with a timeout

Closed

MDEV-14149 galera.MW-328A, galera.MW-328C failed with "Found wrong usage of mutex"

Closed

MDEV-14381 galera.galera_as_master_large, galera.galera_applier_ftwrl_table_alter failed in buildbot with WSREP has not yet prepared node

Closed

MDEV-14496 galera.galera_var_node_address failed in buildbot

Closed

MDEV-15522 Change galera suite MTR tests to use mariabackup instead of xtrabackup

Closed

MDEV-15540 Galera suite MTR tests issuing wsrep_recover fail

Closed

MDEV-15792 Fix mtr to be able to wait for >1 exited mysqld

Closed

MDEV-15794 Test failure on galera.galera_var_retry_autocommit

Closed

MDEV-15803 Test failure on galera.galera_var_auto_inc_control_on

Closed

MDEV-15804 Test failure on galera.pxc-421

Closed

MDEV-15806 Test failure on galera.galera_parallel_simple

Closed

MDEV-15807 Test failure on galera.galera_lock_table

Closed

MDEV-15808 Test failure on galera.galera_gra_log

Closed

MDEV-15809 Test failure on galera.MW-44

Closed

MDEV-15810 Test failure on galera.lp1376747 and galera.lp1376747-2

Closed

MDEV-15811 Test failure on galera.galera_pc_ignore_sb

Closed

MDEV-15823 Test failure on galera.galera_var_slave_threads

Closed

MDEV-15845 Test failure on galera.galera_concurrent_ctas

Closed

MDEV-16008 galera mtr test galera_many_tables_nopk crashed mysqld signal 6

Closed

MDEV-16890 Galera test failure on galera_sst_mysqldump_with_key

Closed

MDEV-17058 Test failure on wsrep.variables

Closed

MDEV-17060 Test failure on galera.galera_var_slave_threads

Closed

MDEV-17061 Test failure on galera.galera_gcs_fc_limit

Closed

MDEV-17108 Test failure on galera.galera_kill_ddl

Closed

MDEV-17143 Galera test failure on galera.MW-44

Closed

MDEV-17151 Galera test failure on galera.galera_var_node_address

Closed

MDEV-17206 Test failure on galera.galera_ist_* in debug builds

Closed

MDEV-17208 Test failure on galera.MW-286

Closed

MDEV-17210 InnoDB: Assertion failure in file lock0lock.cc: 6392

Closed

MDEV-17403 Test failure on galera.galera_enum

Closed

relates to

MDEV-13743 galera_toi_truncate may fail with: query 'reap' succeeded - should have failed with errno 1213

Closed

MDEV-13876 galera.MW-328A failed in buildbot with wrong result or timeout

Closed

MDEV-14069 galera_sst_mysqldump.test fails with: 'show status' failed with wrong errno 1205: 'Lock wait timeout exceeded; try restarting transaction', instead of 0...

Closed

MDEV-14324 Add wsrep_cluster_name=%worker_id%testNo to mtr tests

Closed

MDEV-13738 galera_applier_ftwrl_table_alter may fail with '2006: MySQL server has gone away'

Closed

(39 is blocked by, 5 relates to)

Activity

Ascending order - Click to sort in descending order

View 44 older comments

Jan Lindström (Inactive) added a comment - 2018-02-22 10:28

galera.galera_wsrep_log_conficts 'xtradb' w1 [ fail ]

        Test ended at 2018-02-21 18:41:12

CURRENT_TEST: galera.galera_wsrep_log_conficts

Could not execute 'check-testcase' before testcase 'galera.galera_wsrep_log_conficts' (res: 1):

mysqltest: Logging to '/dev/shm/var/1/tmp/check-mysqld_2.log'.

mysqltest: Results saved in '/dev/shm/var/1/tmp/check-mysqld_2.result'.

mysqltest: Connecting to server localhost:16061 (socket /dev/shm/var/tmp/1/mysqld.2.sock) as 'root', connection 'default', attempt 0 ...

mysqltest: ... Connected.

mysqltest: Start processing test commands from './include/check-testcase.test' ...

mysqltest: At line 78: query 'call mtr.check_testcase()' failed: 1047: WSREP has not yet prepared node for application use

not ok

mysqltest failed but provided no output

Jan Lindström (Inactive) added a comment - 2018-02-22 10:28 galera.galera_wsrep_log_conficts 'xtradb' w1 [ fail ] Test ended at 2018-02-21 18:41:12 CURRENT_TEST: galera.galera_wsrep_log_conficts Could not execute 'check-testcase' before testcase 'galera.galera_wsrep_log_conficts' (res: 1): mysqltest: Logging to '/dev/shm/var/1/tmp/check-mysqld_2.log'. mysqltest: Results saved in '/dev/shm/var/1/tmp/check-mysqld_2.result'. mysqltest: Connecting to server localhost:16061 (socket /dev/shm/var/tmp/1/mysqld.2.sock) as 'root', connection 'default', attempt 0 ... mysqltest: ... Connected. mysqltest: Start processing test commands from './include/check-testcase.test' ... mysqltest: At line 78: query 'call mtr.check_testcase()' failed: 1047: WSREP has not yet prepared node for application use not ok mysqltest failed but provided no output

Marko Mäkelä added a comment - 2019-03-06 07:07

The test galera.MW-336 remains disabled on 10.4 due to result differences like this, on all platforms that run Galera tests:

10.4 2a791c53ad93c8bc1441dd227000234bd49c4990
CURRENT_TEST: galera.MW-336
--- /usr/share/mysql/mysql-test/suite/galera/r/MW-336.result 2019-03-05 10:04:26.000000000 -0500
+++ /dev/shm/var/1/log/MW-336.reject 2019-03-05 14:23:45.776472751 -0500
@@ -13,6 +13,27 @@
COUNT(*)
13
# Wait 9 slave threads to exit 1
+Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+Id User Host db Command Time State Info Progress
+1 system user NULL Sleep 99 wsrep aborter idle NULL 0.000
+2 system user NULL Sleep 99 Waiting on cond NULL 0.000
+3 system user NULL Sleep 63 wsrep applier committed NULL 0.000
+4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
+8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
+95 root localhost test Sleep 32 NULL 0.000
+96 root localhost:42270 test Query 0 Init show full processlist 0.000
+97 system user NULL Sleep 30 wsrep applier idle NULL 0.000
+98 system user NULL Sleep 30 wsrep applier idle NULL 0.000
+99 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+100 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+101 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+102 system user NULL Sleep 30 wsrep applier idle NULL 0.000
+103 system user NULL Sleep 30 wsrep applier idle NULL 0.000
+104 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+105 system user NULL Sleep 30 wsrep applier committed NULL 0.000
SET GLOBAL wsrep_slave_threads = 10;
# Wait 10 slave threads to start 2
SET GLOBAL wsrep_slave_threads = 20;
@@ -25,11 +46,44 @@
COUNT(*)
53
# Wait 10 slave threads to exit 3
+Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+Id User Host db Command Time State Info Progress
+1 system user NULL Sleep 130 wsrep aborter idle NULL 0.000
+2 system user NULL Sleep 130 Waiting on cond NULL 0.000
+4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
+8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
+95 root localhost test Sleep 63 NULL 0.000
+96 root localhost:42270 test Query 0 Init show full processlist 0.000
+115 system user NULL Sleep 30 wsrep applier committed NULL 0.000
SET GLOBAL wsrep_slave_threads = 10;
SET GLOBAL wsrep_slave_threads = 0;
Warnings:
Warning 1292 Truncated incorrect wsrep_slave_threads value: '0'
# Wait 10 slave threads to start 3
+Timeout in wait_condition.inc for SELECT COUNT(*) = 11 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+Id User Host db Command Time State Info Progress
+1 system user NULL Sleep 738 wsrep aborter idle NULL 0.000
+2 system user NULL Sleep 738 Waiting on cond NULL 0.000
+4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
+8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
+95 root localhost test Sleep 671 NULL 0.000
+96 root localhost:42270 test Query 0 Init show full processlist 0.000
+115 system user NULL Sleep 638 wsrep applier committed NULL 0.000
+123 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+116 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+117 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+118 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+119 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+122 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+121 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+120 system user NULL Sleep 608 wsrep applier idle NULL 0.000
+124 system user NULL Sleep 608 wsrep applier idle NULL 0.000
connection node_2;
# Generate 12 replication events
connection node_1;
@@ -37,5 +91,23 @@
COUNT(*)
65
# Wait 10 slave threads to exit 4
+Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%');
+Id User Host db Command Time State Info Progress
+1 system user NULL Sleep 768 wsrep aborter idle NULL 0.000
+2 system user NULL Sleep 768 Waiting on cond NULL 0.000
+4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
+7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
+8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
+95 root localhost test Sleep 701 NULL 0.000
+96 root localhost:42270 test Query 0 Init show full processlist 0.000
+117 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+118 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+119 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+122 system user NULL Sleep 638 wsrep applier idle NULL 0.000
+121 system user NULL Sleep 30 wsrep applier committed NULL 0.000
+120 system user NULL Sleep 638 wsrep applier idle NULL 0.000
+124 system user NULL Sleep 30 wsrep applier committed NULL 0.000
connection node_1;
DROP TABLE t1;

mysqltest: Result length mismatch

Marko Mäkelä added a comment - 2019-03-06 07:07 The test galera.MW-336 remains disabled on 10.4 due to result differences like this, on all platforms that run Galera tests: 10.4 2a791c53ad93c8bc1441dd227000234bd49c4990 CURRENT_TEST: galera.MW-336 --- /usr/share/mysql/mysql-test/suite/galera/r/MW-336.result 2019-03-05 10:04:26.000000000 -0500 +++ /dev/shm/var/1/log/MW-336.reject 2019-03-05 14:23:45.776472751 -0500 @@ -13,6 +13,27 @@ COUNT(*) 13 # Wait 9 slave threads to exit 1 +Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%'); +Id User Host db Command Time State Info Progress +1 system user NULL Sleep 99 wsrep aborter idle NULL 0.000 +2 system user NULL Sleep 99 Waiting on cond NULL 0.000 +3 system user NULL Sleep 63 wsrep applier committed NULL 0.000 +4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 +8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 +95 root localhost test Sleep 32 NULL 0.000 +96 root localhost:42270 test Query 0 Init show full processlist 0.000 +97 system user NULL Sleep 30 wsrep applier idle NULL 0.000 +98 system user NULL Sleep 30 wsrep applier idle NULL 0.000 +99 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +100 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +101 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +102 system user NULL Sleep 30 wsrep applier idle NULL 0.000 +103 system user NULL Sleep 30 wsrep applier idle NULL 0.000 +104 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +105 system user NULL Sleep 30 wsrep applier committed NULL 0.000 SET GLOBAL wsrep_slave_threads = 10; # Wait 10 slave threads to start 2 SET GLOBAL wsrep_slave_threads = 20; @@ -25,11 +46,44 @@ COUNT(*) 53 # Wait 10 slave threads to exit 3 +Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%'); +Id User Host db Command Time State Info Progress +1 system user NULL Sleep 130 wsrep aborter idle NULL 0.000 +2 system user NULL Sleep 130 Waiting on cond NULL 0.000 +4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 +8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 +95 root localhost test Sleep 63 NULL 0.000 +96 root localhost:42270 test Query 0 Init show full processlist 0.000 +115 system user NULL Sleep 30 wsrep applier committed NULL 0.000 SET GLOBAL wsrep_slave_threads = 10; SET GLOBAL wsrep_slave_threads = 0; Warnings: Warning 1292 Truncated incorrect wsrep_slave_threads value: '0' # Wait 10 slave threads to start 3 +Timeout in wait_condition.inc for SELECT COUNT(*) = 11 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%'); +Id User Host db Command Time State Info Progress +1 system user NULL Sleep 738 wsrep aborter idle NULL 0.000 +2 system user NULL Sleep 738 Waiting on cond NULL 0.000 +4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 +8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 +95 root localhost test Sleep 671 NULL 0.000 +96 root localhost:42270 test Query 0 Init show full processlist 0.000 +115 system user NULL Sleep 638 wsrep applier committed NULL 0.000 +123 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +116 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +117 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +118 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +119 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +122 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +121 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +120 system user NULL Sleep 608 wsrep applier idle NULL 0.000 +124 system user NULL Sleep 608 wsrep applier idle NULL 0.000 connection node_2; # Generate 12 replication events connection node_1; @@ -37,5 +91,23 @@ COUNT(*) 65 # Wait 10 slave threads to exit 4 +Timeout in wait_condition.inc for SELECT COUNT(*) = 2 FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER = 'system user' AND (STATE IS NULL OR STATE NOT LIKE 'InnoDB%'); +Id User Host db Command Time State Info Progress +1 system user NULL Sleep 768 wsrep aborter idle NULL 0.000 +2 system user NULL Sleep 768 Waiting on cond NULL 0.000 +4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +5 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +6 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 +7 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 +8 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 +95 root localhost test Sleep 701 NULL 0.000 +96 root localhost:42270 test Query 0 Init show full processlist 0.000 +117 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +118 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +119 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +122 system user NULL Sleep 638 wsrep applier idle NULL 0.000 +121 system user NULL Sleep 30 wsrep applier committed NULL 0.000 +120 system user NULL Sleep 638 wsrep applier idle NULL 0.000 +124 system user NULL Sleep 30 wsrep applier committed NULL 0.000 connection node_1; DROP TABLE t1; mysqltest: Result length mismatch

Marko Mäkelä added a comment - 2019-03-06 07:27

I had disabled galera.MW-336 in MariaDB 10.3.10 due to occasional failures, and recently re-enabled it after 10.3.13 in a merge. I think that the test or code may need to be improved in 10.1‥10.3 as well, in addition to fixing the deterministic failure on 10.4.

Marko Mäkelä added a comment - 2019-03-06 07:27 I had disabled galera.MW-336 in MariaDB 10.3.10 due to occasional failures, and recently re-enabled it after 10.3.13 in a merge . I think that the test or code may need to be improved in 10.1‥10.3 as well, in addition to fixing the deterministic failure on 10.4.

Marko Mäkelä added a comment - 2019-03-08 14:50

I disabled some more tests in 10.1 due to regular failures and plan to merge that all way up to 10.4.

galera.partition and galera.galera_binlog_stmt_autoinc regularly display mismatching values for AUTO_INCREMENT columns, possibly due to nondeterminism in the order in which the nodes are being started or selected for connections.

galera.MW-336 often times out while waiting for something in PROCESSLIST. This might be the same issue as with ~~MDEV-17061~~.

Marko Mäkelä added a comment - 2019-03-08 14:50 I disabled some more tests in 10.1 due to regular failures and plan to merge that all way up to 10.4. galera.partition and galera.galera_binlog_stmt_autoinc regularly display mismatching values for AUTO_INCREMENT columns, possibly due to nondeterminism in the order in which the nodes are being started or selected for connections. galera.MW-336 often times out while waiting for something in PROCESSLIST. This might be the same issue as with MDEV-17061 .

Jan Lindström (Inactive) added a comment - 2019-07-03 12:40

Closing as we have a new ~~MDEV-19937~~.

Jan Lindström (Inactive) added a comment - 2019-07-03 12:40 Closing as we have a new MDEV-19937 .

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Jan Lindström (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2017-08-16 08:31

Updated:: 2019-12-18 16:05

Resolved:: 2019-07-03 12:40

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration