[MDEV-19983] Galera: Rolling upgrade: Upgraded node 2 cannot connect to the cluster on rows adding, updating, and removing during upgrade Created: 2019-07-08  Updated: 2022-08-30  Resolved: 2019-09-23

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.7
Fix Version/s: 10.4.9

Type: Bug Priority: Critical
Reporter: Stepan Patryshev (Inactive) Assignee: Stepan Patryshev (Inactive)
Resolution: Fixed Votes: 0
Labels: galera_4
Environment:

CentOS Linux release 7.6.1810 (Core)

Attachments: File my.cnf     File mysqld.1.err     File mysqld.2.cnf     File mysqld.2.err     File mysqld.3.cnf     File mysqld.3.err     Text File stdout.log    
Issue Links:
Relates
relates to MDEV-29246 WSREP_CLUSTER_SIZE at 0 after rolling... Closed
relates to MDEV-18271 Galera 4: test manually rolling upgra... Closed

 Description   


Galera: Rolling upgrade: Upgraded node 2 cannot connect to the cluster on rows adding, updating, and removing during upgrade.

This issue was discovered on testing of Rolling Upgrade according to "MariaDB 10.4 Cluster Rolling Upgrade - Naive Approach" by Seppo Jaakola: https://docs.google.com/document/d/1z4XTpLpzStWMFaNnrSmiESaIVeCoKhu9Hbb1SrDPf0w

All binaries are non-debug built from sources:

MariaDB Server 10.4: branch 10.4, commit 9d6b601e797dd8333340dadaefae09ebafc787db.
Galera Lib4: branch mariadb-4.x, commit ba337dd0ac281a5e9f29c652a890bd7ad2ac464e.

MariaDB Server 10.3: branch 10.3, commit 099007c3c92d1405625777fa86d2fba3da1d339c.
Galera Lib3: branch mariadb-3.x, commit 227e96e457acb60037450bc1e81c45594782e906.

Steps:

1. Start 3 MariaDB 10.3 nodes with mtr:
1.0. export WSREP_PROVIDER=/usr/lib/libgalera_smm_3.so
1.1. cd mysql-test
1.2. "./mtr --suite=galera_3nodes --start-and-exit"

NODE 3 UPGRADE:

2. Copy [mysqld.3] group from var/my.cnf (attached my.cnf) into separate configuration file: mysqld.3.cnf (attached mysqld.3.cnf), and make following edits:

2.1. Edit:

[mysqld]
 
wsrep_cluster_address='gcomm://127.0.0.1:16003,127.0.0.1:16006,127.0.0.1:16009'
 
wsrep_provider=<path to galera 4 library>
 
basedir=<10.4 source tree>
 
character-sets-dir=<10.4 source tree>/sql/share/charsets
 
lc-messages-dir=<10.4 source tree>/sql/share/

2.2. And add there also:

binlog-format=row
 
wsrep_sst_method=rsync
 
innodb-autoinc-lock-mode=2

3. Load test data.

3.1. Run the client:
/home/stepan/mariadb/10.3/client/mysql -u root -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.1.sock

3.2. Create test table:
use test;
create table t (i int primary key auto_increment, j int);

3.3. Load some data for each node:
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16000 -e'insert into test.t(j) values(111)' "
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16001 -e'insert into test.t(j) values(222)' "
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16002 -e'insert into test.t(j) values(333)' "

4. Upgrade node 3.

4.1. Stop data loading for the node 3 (-P16002).

4.2 Stop the Server on node 3:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.3.sock

4.3. Remove some rows on the node 1:
delete from t where i < 10;

4.4. Update a row on the node 1:
update t set j = 10001 where i = 10;

4.5. Make sure that wsrep-on is off:
vi /home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf
#wsrep-on=1

4.6. Run 10.4 binaries with 10.3 data:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf --wsrep_provider=none

4.7. Run mysql_upgrade:
/home/stepan/galera/git/10.4/server/client/mysql_upgrade --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf -uroot -h0 -P16002

4.8. Stop the Server:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.3.sock

4.9. export PATH=$PATH:/home/stepan/galera/git/10.4/server/scripts

5. Check upgraded node 3 without the cluster.

5.1. Start the server:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf

5.2. Start the client:
/home/stepan/mariadb/10.3/client/mysql -u root -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.3.sock

Actual result:
Server version: 10.4.7-MariaDB-log Source distribution

5.3. Stop the Server:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.3.sock

6. Join node 3 back to the cluster.

6.1. Add to /home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf:

wsrep-on=1

6.2. Start the server:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.3.cnf

6.3. Check the ports:

$ sudo netstat -tulpn | grep mysqld
 
tcp        0      0 0.0.0.0:16006           0.0.0.0:*               LISTEN      7328/mysqld
 
tcp        0      0 0.0.0.0:16009           0.0.0.0:*               LISTEN      10501/mysqld
 
tcp        0      0 127.0.0.1:16000         0.0.0.0:*               LISTEN      7327/mysqld
 
tcp        0      0 127.0.0.1:16001         0.0.0.0:*               LISTEN      7328/mysqld
 
tcp        0      0 127.0.0.1:16002         0.0.0.0:*               LISTEN      10501/mysqld
 
tcp        0      0 0.0.0.0:16003           0.0.0.0:*               LISTEN      7327/mysqld

6.4. Start the client:
/home/stepan/mariadb/10.3/client/mysql -u root -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.3.sock

6.5. Check WSREP cluster status:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_%';
 
+----------------------------+----------------------+
 
| Variable_name              | Value                |
 
+----------------------------+----------------------+
 
| wsrep_cluster_weight       | 3                    |
 
| wsrep_cluster_capabilities |                      |
 
| wsrep_cluster_conf_id      | 18446744073709551615 |
 
| wsrep_cluster_size         | 0                    |
 
| wsrep_cluster_state_uuid   |                      |
 
| wsrep_cluster_status       | Primary              |
 
+----------------------------+----------------------+

6.6. Check the presence of the new 10.4 tables:

use mysql;
 
show tables LIKE 'wsrep_%';
 
+---------------------------+
 
| Tables_in_mysql (wsrep_%) |
 
+---------------------------+
 
| wsrep_cluster             |
 
| wsrep_cluster_members     |
 
| wsrep_streaming_log       |
 
+---------------------------+

Actual result:
Upgraded node 3 has joined the cluster successfully.

NODE 2 UPGRADE:

7. Create [mysqld.2] the same way as described on the step 2.

8.1. Load some data for each node:
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16000 -e'insert into test.t(j) values(111)' "
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16001 -e'insert into test.t(j) values(222)' "
watch "/home/stepan/mariadb/10.3/client/mysql -uroot -h0 -P16002 -e'insert into test.t(j) values(333)' "

9. Upgrade node 2.

9.1. Stop data loading for the node 2 (-P16001).

9.2 Stop the Server on node 2:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock

9.3. Remove some rows on the node 1:
delete from t where i < 20;

9.4. Update a row on the node 1:
update t set j = 20002 where i = 20;

9.5. Make sure that wsrep-on is off:
vi /home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf
#wsrep-on=1

9.6. Run 10.4 binaries with 10.3 data:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf --wsrep_provider=none

9.7. Run mysql_upgrade:
/home/stepan/galera/git/10.4/server/client/mysql_upgrade --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf -uroot -h0 -P16001

9.8. Stop the Server:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock

9.9. export PATH=$PATH:/home/stepan/galera/git/10.4/server/scripts

10. Check upgraded node 2 without the cluster.

10.1. Start the server:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf

10.2. Start the client:
/home/stepan/mariadb/10.3/client/mysql -u root -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock
Actual result:
Server version: 10.4.7-MariaDB-log Source distribution

10.3. Stop the Server:
/home/stepan/mariadb/10.3/client/mysqladmin -u root shutdown -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock

11. Join node 2 back to the cluster.

11.1. Add to /home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf:

wsrep-on=1

11.2. Start the server:
/home/stepan/galera/git/10.4/server/sql/mysqld --defaults-file=/home/stepan/mariadb/10.3/mysql-test/var/mysqld.2.cnf

Actual results:

11.3. The port 16006 is not used:

$ sudo netstat -tulpn | grep mysqld
 
tcp        0      0 0.0.0.0:16009           0.0.0.0:*               LISTEN      10501/mysqld
 
tcp        0      0 127.0.0.1:16000         0.0.0.0:*               LISTEN      7327/mysqld
 
tcp        0      0 127.0.0.1:16001         0.0.0.0:*               LISTEN      14576/mysqld
 
tcp        0      0 127.0.0.1:16002         0.0.0.0:*               LISTEN      10501/mysqld
 
tcp        0      0 0.0.0.0:16003           0.0.0.0:*               LISTEN      7327/mysqld

11.4. Run the client on node 2:
/home/stepan/mariadb/10.3/client/mysql -u root -S /home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock

11.5. Node 2 is NOT connected to the cluster:

SHOW GLOBAL STATUS LIKE 'wsrep_cluster_%';
 
+----------------------------+--------------------------------------+
 
| Variable_name              | Value                                |
 
+----------------------------+--------------------------------------+
 
| wsrep_cluster_capabilities |                                      |
 
| wsrep_cluster_conf_id      | 18446744073709551615                 |
 
| wsrep_cluster_size         | 0                                    |
 
| wsrep_cluster_state_uuid   | e3c49d03-a155-11e9-80eb-a614fcadb109 |
 
| wsrep_cluster_status       | Disconnected                         |
 
+----------------------------+--------------------------------------+

11.6. Data from the test table cannot be selected: 

select * from t;
 
ERROR 1047 (08S01): WSREP has not yet prepared node for application use

11.7. mysqld.2.err: [ERROR] WSREP: Failed to JOIN the cluster after SST:

2019-07-08 11:49:36 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2703)
 
2019-07-08 11:49:36 3 [Note] WSREP: Requesting state transfer: success, donor: 0
 
2019-07-08 11:49:36 0 [ERROR] WSREP: got asio system error while reading IST stream: asio.system:104
 
2019-07-08 11:49:36 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 2703 last received: -1
 
[...]
 
2019-07-08 11:49:38 4 [Warning] WSREP: View recovered from stable storage was empty. If the server is doing rolling upgrade from previous version which does not support storing view info into stable storage, this is ok. Otherwise this may be a sign of malfunction.
 
2019-07-08 11:49:38 12 [Note] WSREP: Cluster table is empty, not recovering transactions
 
2019-07-08 11:49:38 4 [Note] WSREP: SST received: e3c49d03-a155-11e9-80eb-a614fcadb109:1523
 
2019-07-08 11:49:38 3 [Warning] WSREP: moving position backwards: 2703 -> 1523
 
[...]
 
2019-07-08 11:49:38 0 [Note] /home/stepan/galera/git/10.4/server/sql/mysqld: ready for connections.
 
Version: '10.4.7-MariaDB-log'  socket: '/home/stepan/mariadb/10.3/mysql-test/var/tmp/mysqld.2.sock'  port: 16001  Source distribution
 
2019-07-08 11:49:38 3 [Note] WSREP: Receiving IST: 1180 writesets, seqnos 1524-2703
 
2019-07-08 11:49:38 3 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
 
	 at galera/src/replicator_smm.hpp:pop_front():320. Null event.
 
[...]
 
2019-07-08 11:49:38 3 [Note] WSREP: IST received: e3c49d03-a155-11e9-80eb-a614fcadb109:-1
 
2019-07-08 11:49:38 3 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
 
2019-07-08 11:49:38 3 [ERROR] WSREP: Failed to JOIN the cluster after SST

11.8. mysqld.3.err:

2019-07-08 11:49:36 0 [Note] WSREP: async IST sender starting to serve tcp://127.0.0.1:16007 sending 1524-157
 
2019-07-08 11:49:36 0 [ERROR] WSREP: async IST sender failed to serve tcp://127.0.0.1:16007: sender send first greater than last: 1524 > 157: 22 (Invalid argument)
 
	 at galera/src/ist.cpp:send():783

Expected result:
Upgraded node 2 successfully connects to the cluster on rows adding, updating, and removing during upgrade.

Other logs are also attached.

 Comments   
Comment by Shahriyar Rzayev (Inactive) [ 2019-07-08 ]

Hi Stepan,
Do you have a more simple case to reproduce? Simple MTR will be great.
Btw, I have spotted another issue, not this one but again with rsync, will open a separate ticket for that.

Comment by Stepan Patryshev (Inactive) [ 2019-07-08 ]

Just have reproduced it without adding or removing any rows, but only with updating one row.

Comment by Stepan Patryshev (Inactive) [ 2019-07-08 ]

shako pfb more simple steps:

Rolling upgrade from 10.3 to 10.4.

1. Upgrade and join the cluster node 3.
2. Stop node 2.
3. Update one row in the cluster (on node 1 or 3).
4. Upgrade node 2.
5. Try to join upgraded node 2 to the cluster.

Comment by Shahriyar Rzayev (Inactive) [ 2019-07-09 ]

Reproduced as described. Also sysbench load can help to create write sets.
I think the main problem is on upgraded node.
Steps:

  • Start 3 nodes 10.3 + Galera 3
  • Upgrade node3 -> 10.4 + Galera 4
  • Stop node2
  • Do some updates on node3
  • Upgrade node2 - try to start
  • Lose node3 as:

2019-07-09 12:25:05 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '127.0.0.1:6446/rsync_sst' --socket '/tmp/mysql-node3.sock' --datadir '/home
 
/shako/Galera_Tests/datadirs/node3/'  --defaults-file '/home/shako/Galera_Tests/datadirs/mynode3_10.4.cnf'   --binlog 'QA'   '' --gtid '24b4b4cc-a221-11e9-a749-9a0
 
b9d0d98e2:2550' --gtid-domain-id '-1' --bypass'
 
2019-07-09 12:25:05 3 [Note] WSREP: sst_donor_thread signaled with 0
 
2019-07-09 12:25:05 0 [Note] WSREP: async IST sender starting to serve tcp://127.0.0.1:5568 sending 2551-0
 
mysqld: /home/shako/Galera_Tests/Galera/galera/src/ist.cpp:782: void galera::ist::Sender::send(wsrep_seqno_t, wsrep_seqno_t, wsrep_seqno_t): Assertion `0' failed.
 
190709 12:25:05 [ERROR] mysqld got signal 6 ;


(gdb) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
 
#1  0x00007f6d098dc801 in __GI_abort () at abort.c:79
 
#2  0x00007f6d098cc39a in __assert_fail_base (fmt=0x7f6d09a537d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f6d08592d1d "0", 
    file=file@entry=0x7f6d0859f830 "/home/shako/Galera_Tests/Galera/galera/src/ist.cpp", line=line@entry=782, 
    function=function@entry=0x7f6d085a03a0 <galera::ist::Sender::send(long, long, long)::__PRETTY_FUNCTION__> "void galera::ist::Sender::send(wsrep_seqno_t, wsrep_seqno_t, wsrep_seqno_t)") at assert.c:92
 
#3  0x00007f6d098cc412 in __GI___assert_fail (assertion=assertion@entry=0x7f6d08592d1d "0", 
    file=file@entry=0x7f6d0859f830 "/home/shako/Galera_Tests/Galera/galera/src/ist.cpp", line=line@entry=782, 
    function=function@entry=0x7f6d085a03a0 <galera::ist::Sender::send(long, long, long)::__PRETTY_FUNCTION__> "void galera::ist::Sender::send(wsrep_seqno_t, wsrep_seqno_t, wsrep_seqno_t)") at assert.c:101
 
#4  0x00007f6d0853f939 in galera::ist::Sender::send (this=this@entry=0x7f6cdc043100, first=54, last=last@entry=0, preload_start=preload_start@entry=-1)
 
    at /home/shako/Galera_Tests/Galera/galera/src/ist.cpp:782
 
#5  0x00007f6d0853ffd6 in run_async_sender (arg=0x7f6cdc043100) at /home/shako/Galera_Tests/Galera/galera/src/ist.cpp:907
 
#6  0x00007f6d0a7d76db in start_thread (arg=0x7f6c477fe700) at pthread_create.c:463
 
#7  0x00007f6d099bd88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Shahriyar Rzayev (Inactive) [ 2019-07-09 ]

Same issue with mariabackup sst method.
In my tests, I have added DDL/DMLs to load other nodes during upgrade, but missed the same thing between server shutdowns.
Say we have upgraded node3 to 10.4, and now we shutdown node2 to upgrade it - several statements should be run on the upgraded node3. Added to test scenario list.

With mariabackup from upgraded node3 we got:

2019-07-09 13:25:10 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '127.0.0.1:6444/xtrabackup_sst//1' --socket '/tmp/mysql-node3.sock' --
 
datadir '/home/shako/Galera_Tests/datadirs/node3/'  --defaults-file '/home/shako/Galera_Tests/datadirs/mynode3_10.4.cnf'   --binlog 'QA'   '' --gtid '2a903ca9-a22b
 
-11e9-a4e6-520593c25982:61' --gtid-domain-id '-1' --bypass'
 
2019-07-09 13:25:10 2 [Note] WSREP: sst_donor_thread signaled with 0
 
2019-07-09 13:25:10 0 [Note] WSREP: async IST sender starting to serve tcp://127.0.0.1:5568 sending 62-0
 
mysqld: /home/shako/Galera_Tests/Galera/galera/src/ist.cpp:782: void galera::ist::Sender::send(wsrep_seqno_t, wsrep_seqno_t, wsrep_seqno_t): Assertion `0' failed.
 
190709 13:25:10 [ERROR] mysqld got signal 6 ;

Trying to start/upgrade node2 and failed as:

2019-07-09 13:25:11 3 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
 
         at /home/shako/Galera_Tests/Galera/galera/src/replicator_smm.hpp:pop_front():320. Null event.
 
mysqld: /home/shako/Galera_Tests/Galera/galera/src/replicator_str.cpp:955: void galera::ReplicatorSMM::request_state_transfer(void*, const wsrep_uuid_t&, wsrep_seqno_t, const void*, ssize_t): Assertion `ist_seqno > sst_seqno_' failed.
 
190709 13:25:20 [ERROR] mysqld got signal 6 ;
 
src/replicator_str.cpp:965(galera::ReplicatorSMM::request_state_transfer(void*, wsrep_uuid const&, long, void const*, long))[0x7fd32c7213ad]
 
src/replicator_smm.cpp:2662(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7fd32c709b4e]
 
src/gcs_action_source.cpp:124(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7fd32c6f1c98]
 
src/gcs_action_source.cpp:182(galera::GcsActionSource::process(void*, bool&))[0x7fd32c6f1fdf]
 
src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7fd32c7085a6]
 
src/wsrep_provider.cpp:263(galera_recv)[0x7fd32c72451e]

Comment by Stepan Patryshev (Inactive) [ 2019-07-09 ]

It's reproduced also just on adding rows without removing or updating them.

Comment by Stepan Patryshev (Inactive) [ 2019-07-10 ]

It's reproduced also on Enterprise Server:

MariaDB Enterprise Server 10.4: branch 10.4-enterprise, commit 90fc8f111b41a5c0f10c430355cf905f76658161.
Enterprise Galera Lib4: branch es-mariadb-4.x, commit f1da6733adf409e6b5bf020c123ae74d55085fad.

MariaDB Enterprise Server 10.3: 10.3-enterprise, commit 492555cbabbd79e8554ef41239b64bd497b44764.
Enterprise Galera Lib3: branch es-mariadb-3.x, commit c9233d02532d138a0e68be208795a8837d526af7.

Comment by Stepan Patryshev (Inactive) [ 2019-07-11 ]

But it's not reproduced during rolling upgrade from ES 10.2 + Enterprise Galera 3 to ES 10.3 + Enterprise Galera 3:
MariaDB Enterprise Server 10.2: branch 10.2-enterprise, commit 283531e2028e8550a92c1de0bf3926759f944257.
MariaDB Enterprise Server 10.3: 10.3-enterprise, commit 492555cbabbd79e8554ef41239b64bd497b44764.
Enterprise Galera Lib3: branch es-mariadb-3.x, commit c9233d02532d138a0e68be208795a8837d526af7.

Comment by Stepan Patryshev (Inactive) [ 2019-09-20 ]

It's not reproduced during rolling upgrade from CS 10.3 to 10.4 on the following versions:
All binaries are non-debug built from sources:

MariaDB Server 10.4.9-MariaDB-debug: branch 10.4, commit b9dea911bf8e3d4d8fc57ce3ef15ab0e2ab4d076.
Galera Lib4 26.4.3(r4535): branch mariadb-4.x, commit 752664dc3c7065d8e0c73ac99d0028a5f84eb250.

MariaDB Server 10.3.19-MariaDB-debug: branch 10.3, commit 90a9c4cae74d2ef1008e3f216026b7fd2697e46b.
Galera Lib3 25.3.28(r3875): branch mariadb-3.x, commit 92d6990b65259d12327c211e6658cf1c0c818cfb.

Comment by Stepan Patryshev (Inactive) [ 2019-09-23 ]

It's not reproduced during rolling upgrade also from ES 10.3 to 10.4 on the following versions:
All binaries are non-debug built from sources:

MariaDB Enterprise Server 10.3.18-3-MariaDB-debug, branch 10.3-enterprise, commit 83eedda61ffd90233182af6f4db89ecd646fb7d5.
Enterprise Galera Lib3: branch es-mariadb-3.x, commit 9837a754cf5e6dc55322ef41f01512643a002f8c.

MariaDB Enterprise Server 10.4.8-3-MariaDB-debug: branch 10.4-enterprise, commit e6bf0196213eb4d3dceecc5955fb7cadc6e3929e.
Enterprise Galera Lib4: branch es-mariadb-4.x, commit fbd94378ec7fbc97984848db0f24777b673d4319.

Comment by Stepan Patryshev (Inactive) [ 2019-09-23 ]

Closing it as fixed.

Generated at Thu Feb 08 08:55:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.