Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Duplicate
-
10.4.26
-
Debian Buster in a 5 node Galera cluster
Description
Since upgrading MariaDB to 10.4.26 our mariabackup processes don't finish, but simply run into a loop:
[01] 2022-08-24 15:48:06 ...done
[00] 2022-08-24 15:48:07 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:08 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:09 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:10 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:11 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:12 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:13 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:14 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:15 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:16 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:17 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:18 >> log scanned up to (377301702)
2022-08-24 15:48:19 0 [Note] InnoDB: Read redo log up to LSN=377302016
Here is the command I run:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info
Seems like Acquiring BACKUP LOCKS... step isn't happening in the new version with full backup.
Not seeing this output in my logs with the new version like I do with previous version 10.4.25:
2022-08-24 15:59:34 62 [Note] WSREP: Desyncing and pausing the provider
2022-08-24 15:59:34 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) desyncs itself from group
2022-08-24 15:59:34 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 68)
2022-08-24 15:59:34 62 [Note] WSREP: pause
2022-08-24 15:59:34 62 [Note] WSREP: Provider paused at aadbd2c9-f7a7-11ec-8153-d6a691037b36:68 (6)
2022-08-24 15:59:34 62 [Note] WSREP: Provider paused at: 68
2022-08-24 15:59:35 62 [Note] WSREP: Resuming and resyncing the provider
2022-08-24 15:59:35 62 [Note] WSREP: resume
2022-08-24 15:59:35 62 [Note] WSREP: resuming provider at 6
2022-08-24 15:59:35 62 [Note] WSREP: Provider resumed.
2022-08-24 15:59:35 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) resyncs itself to group.
2022-08-24 15:59:35 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 68)
2022-08-24 15:59:35 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete.
2022-08-24 15:59:35 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) synced with group.
2022-08-24 15:59:35 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2022-08-24 15:59:35 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 68)
2022-08-24 15:59:35 12 [Note] WSREP: Server mwpdbbackup-staging14 synced with group
I can do incremental backup on an existing full backup successfully, but I cannot create a new full backup
When I run incremental, I use the following command:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info --incremental-lsn=377305294
and I see the host desync itself and successfully complete the backup.
[00] 2022-08-24 15:59:34 >> log scanned up to (377305303)
[00] 2022-08-24 15:59:34 Acquiring BACKUP LOCKS...
[00] 2022-08-24 15:59:34 Starting to backup non-InnoDB tables and files
When I specify a full backup to be taken, using this command, it loops through the LSN reads:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info
and I don't see it desyncing from the cluster
It looks like it doesn't finish the step to move onto non-innodb tables and files.
I've tried turning the innodb_instances to 1 (as described by this bug report in case it was related) but it doesn't resolve the issue.
Attachments
Issue Links
- duplicates
-
MDEV-29043 mariabackup --compress --parallel=4 hangs in current 10.5.17 from GitHub
- Closed