We've upgraded from MariaDB 10.3 to the latest version of 10.4 (10.4.7). Once the upgrade has been completed we noticed some strange behavior during cluster-initialisation. No matter which node of our 7-node-cluster we start, it takes approx 8 minutes to get the node into the cluster, each time. Another test-cluster of ours has also inreased joining and sync times, but i'll stick to our production cluster here.
The first time a "hung up" occurs is at the following point (see attached log):
Aug 7 12:59:36 MariaDB-PROD-001 mysqld: 2019-08-07 12:59:36 2 [Note] WSREP: ####### drain monitors upto -1
Getting past this point takes 4 minutes and 33 seconds (273 seconds)
The second time during the join-phase the problem occurs here:
Aug 7 13:04:10 MariaDB-PROD-001 mysqld: 2019-08-07 13:04:10 0 [Note] WSREP: ####### drain monitors upto 0
Aug 7 13:04:10 MariaDB-PROD-001 systemd: Started MariaDB 10.4.7 database server.
Aug 7 13:04:12 MariaDB-PROD-001 mysqld: 2019-08-07 13:04:12 0 [Note] InnoDB: Buffer pool(s) load completed at 190807 13:04:12
Aug 7 13:08:43 MariaDB-PROD-001 mysqld: 2019-08-07 13:08:43 0 [Note] WSREP: REPL Protocols: 10 (5, 3)
This one took 4 minutes and 31 seconds (271 seconds), nearly exactly like the other one before o_Ô. So this might be something to work keep an eye on?!
Same problems apply to all and every of our 7 node Cluster, also our 2nd cluster is affected.