This is most probably result of fixes related to MDEV-23080 as this behavior appeared after updating to version 10.4.21.
We run galera cluster of 3 nodes. On every backup performed with mariabackup after update we get "Member desyncs itself from group" during backup final phase.
Mariabackup is started with:
mariabackup -u root -p PASSWORD --backup --galera-info --stream=xbstream --parallel 8 --use-memory=16G --socket=/var/run/mysqld/mysqld.sock --datadir=/var/lib/mysql 2>>/var/log/mariabackup_copy.log| /usr/bin/zstd --fast -T8 -q -o /home/mariabackup/backup.zst
|
While creating the backup, after phase of streaming InnoDB data, phase of streaming non-InnoDB data comes, for example:
mariabackup log:
[00] 2021-08-17 02:54:31 Acquiring BACKUP LOCKS...
|
[00] 2021-08-17 02:54:34 Starting to backup non-InnoDB tables and files
|
...
|
[00] 2021-08-17 02:57:53 Finished backing up non-InnoDB tables and files
|
[00] 2021-08-17 02:57:53 Waiting for log copy thread to read lsn 32481906458304
|
...
|
[00] 2021-08-17 02:57:56 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
|
[00] 2021-08-17 02:57:56 mariabackup: The latest check point (for incremental): '32481906568593'
|
[00] 2021-08-17 02:57:56 Executing BACKUP STAGE END
|
[00] 2021-08-17 02:57:56 All tables unlocked
|
[00] 2021-08-17 02:57:56 Streaming ib_buffer_pool to <STDOUT>
|
[00] 2021-08-17 02:57:56 Backup created in directory '/xtrabackup_backupfiles/'
|
[00] 2021-08-17 02:57:56 MySQL binlog position: filename 'mariadb-bin.019684', position '421', GTID of the last change ''
|
[00] 2021-08-17 02:57:56 Streaming backup-my.cnf
|
[00] 2021-08-17 02:57:56 Streaming xtrabackup_info
|
[00] 2021-08-17 02:57:56 Redo log (from LSN 32481655310193 to 32481906568602) was copied.
|
[00] 2021-08-17 02:57:56 completed OK!
|
Within this phase, .TRG, .PAR, .FRM and other metadata files are copied. But after last update node started to report self-desync immediately when backup comes to this phase.
Node remains desynced until backup finished and then synchronizes with others.
mysqld log:
2021-08-17 2:54:34 18009831 [Note] WSREP: Desyncing and pausing the provider
|
2021-08-17 2:54:34 0 [Note] WSREP: Member 0.0 (node2.localdomain) desyncs itself from group
|
2021-08-17 2:54:34 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3821784561)
|
2021-08-17 2:54:34 18009831 [Note] WSREP: pause
|
2021-08-17 2:54:34 18009831 [Note] WSREP: Provider paused at 0ca12340-****-****-****-******ed4dfb:3821784561 (30463042)
|
2021-08-17 2:54:34 18009831 [Note] WSREP: Provider paused at: 3821784561
|
2021-08-17 2:57:56 18009831 [Note] WSREP: Resuming and resyncing the provider
|
2021-08-17 2:57:56 18009831 [Note] WSREP: resume
|
2021-08-17 2:57:56 18009831 [Note] WSREP: resuming provider at 30463042
|
2021-08-17 2:57:56 18009831 [Note] WSREP: Provider resumed.
|
2021-08-17 2:57:56 0 [Note] WSREP: Member 0.0 (node2.localdomain) resyncs itself to group.
|
2021-08-17 2:57:56 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 3821789518)
|
2021-08-17 2:57:57 0 [Note] WSREP: Member 0.0 (node2.localdomain) synced with group.
|
2021-08-17 2:57:57 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3821789533)
|
2021-08-17 2:57:57 24 [Note] WSREP: Server node2.localdomain synced with group
|
Note desync event just after starting to stream non-InnoDB files!
As dataset includes significant number of databases (just several thousands), phase of streaming non-InnoDB data can take several minutes, real lentgh of this desynced phase depends on number of tables the cluster handles, can be even longer than what we hit. For all this time the node remains desynced.
There are two questions:
- Is it really necessary to put node into Desynced state while streaming non-InnoDB data (considering the fact that WSREP only replicates InnoDB transactions) and
- is there any safe workaround on this that wouldn't render backup into inconsistent set of data?
We use mariabackup for long time already, but there was no such a behavior before.
Have to say that this issue neutralizes main mariabackup advantage of being non-blocking and artificially decreases cluster availability. Can this be fixed without breaking previous fixes for MDEV-23080?
MDEV-23080 mariabackup: position saved in xtrabackup_binlog_info is incorrect for replication
