Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Duplicate
-
10.4.26
-
Debian Buster in a 5 node Galera cluster
Description
Since upgrading MariaDB to 10.4.26 our mariabackup processes don't finish, but simply run into a loop:
[01] 2022-08-24 15:48:06 ...done
[00] 2022-08-24 15:48:07 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:08 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:09 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:10 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:11 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:12 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:13 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:14 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:15 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:16 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:17 >> log scanned up to (377301702)
[00] 2022-08-24 15:48:18 >> log scanned up to (377301702)
2022-08-24 15:48:19 0 [Note] InnoDB: Read redo log up to LSN=377302016
Here is the command I run:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info
Seems like Acquiring BACKUP LOCKS... step isn't happening in the new version with full backup.
Not seeing this output in my logs with the new version like I do with previous version 10.4.25:
2022-08-24 15:59:34 62 [Note] WSREP: Desyncing and pausing the provider
2022-08-24 15:59:34 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) desyncs itself from group
2022-08-24 15:59:34 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 68)
2022-08-24 15:59:34 62 [Note] WSREP: pause
2022-08-24 15:59:34 62 [Note] WSREP: Provider paused at aadbd2c9-f7a7-11ec-8153-d6a691037b36:68 (6)
2022-08-24 15:59:34 62 [Note] WSREP: Provider paused at: 68
2022-08-24 15:59:35 62 [Note] WSREP: Resuming and resyncing the provider
2022-08-24 15:59:35 62 [Note] WSREP: resume
2022-08-24 15:59:35 62 [Note] WSREP: resuming provider at 6
2022-08-24 15:59:35 62 [Note] WSREP: Provider resumed.
2022-08-24 15:59:35 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) resyncs itself to group.
2022-08-24 15:59:35 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 68)
2022-08-24 15:59:35 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete.
2022-08-24 15:59:35 0 [Note] WSREP: Member 3.0 (mwpdbbackup-staging14) synced with group.
2022-08-24 15:59:35 0 [Note] WSREP: Processing event queue:...100.0% (1/1 events) complete.
2022-08-24 15:59:35 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 68)
2022-08-24 15:59:35 12 [Note] WSREP: Server mwpdbbackup-staging14 synced with group
I can do incremental backup on an existing full backup successfully, but I cannot create a new full backup
When I run incremental, I use the following command:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info --incremental-lsn=377305294
and I see the host desync itself and successfully complete the backup.
[00] 2022-08-24 15:59:34 >> log scanned up to (377305303)
[00] 2022-08-24 15:59:34 Acquiring BACKUP LOCKS...
[00] 2022-08-24 15:59:34 Starting to backup non-InnoDB tables and files
When I specify a full backup to be taken, using this command, it loops through the LSN reads:
mariabackup --defaults-file=/etc/mysql/backup.cnf --extra-lsndir=/var/lib/mysql_backups/2022-08-24 --backup --compress --stream=xbstream --binlog-info=ON --parallel=2 --compress-threads=2 --galera-info
and I don't see it desyncing from the cluster
It looks like it doesn't finish the step to move onto non-innodb tables and files.
I've tried turning the innodb_instances to 1 (as described by this bug report in case it was related) but it doesn't resolve the issue.
Attachments
Issue Links
- duplicates
-
MDEV-29043 mariabackup --compress --parallel=4 hangs in current 10.5.17 from GitHub
-
- Closed
-
By design, mariabackup uses a dedicated thread that keeps copying or polling the InnoDB redo log for any new changes. Concurrently with this thread, any data files will be copied.
The log-copying thread is not supposed to terminate until the very end. If there is no file copying activity going on, then the progress should actually be blocked by the server. At the end, mariabackup will issue a command like FLUSH TABLES WITH READ LOCK or BACKUP STAGE BLOCK_COMMIT. Depending on the operations that are concurrently executing on the server, it may take a long time for such a command to complete.
Please check what is happening on the server side, by issuing a command like SHOW PROCESSLIST.