[MDEV-23464] mariabackup 10.3.21 fails every time since upgrading from 10.2.14 Created: 2020-08-12 Updated: 2023-04-16 Resolved: 2023-04-16 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Backup, mariabackup |
| Affects Version/s: | 10.3.24 |
| Fix Version/s: | N/A |
| Type: | Bug | Priority: | Major |
| Reporter: | Will Mayall | Assignee: | Marko Mäkelä |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | crash, upgrade | ||
| Environment: |
10.3.16, 10.3.21, 10.4.13, Centos 6, ext4 |
||
| Attachments: |
|
| Description |
|
Since upgrading to 10.3, I have not been able to get my Production databases backed up using mariabackup. I have 40 servers, 1TB to 30TB, most servers have thousands of identical databases. Here is a snipnet of one failure, however mariabackup has failed on other tables, and I am able to do a select * from the table without a crash. The backup fails about an hour to 2 hours into the backup. One thing in common with all the tables reporting corruption either have Data Types of TEXT or LONGTEXT. I also lftp'ed one image to a test and upgraded it to 10.4.13. [03] 2020-08-07 22:57:50 Database page corruption detected at page 32425, retrying... The command I used: |
| Comments |
| Comment by Will Mayall [ 2020-09-12 ] | |||||||||||||
|
I found a viable but ugly work around. Will Mayall | |||||||||||||
| Comment by Vladislav Vaintroub [ 2021-05-10 ] | |||||||||||||
|
what's your workaround? | |||||||||||||
| Comment by Will Mayall [ 2021-05-10 ] | |||||||||||||
|
I am running the mariabackup from a Production Server, I don't care if it takes mariabackup more time to complete than cause issue to my server (increased load), therefore I limit the resources mariabackup can use. I have been working on MySQL for around 20 years, I haven't had an issue using --use-memory with xtrabackup, the error doesn't indicate that --use-memory is the issue. I have tried many combinations with xbstream and mbstream, same results, I have to uncompress the tables that fail for corruption, then the backup completes. Again, I have around 100 Servers, each server has 2K databases with the same schemas, the data is different. mariabackup has failed on every server I have tried, but the corruption doesn't happen on every table, maybe 10 to 20 tables, and it is only ONE Table Definition that mariabackup fails on. CREATE TABLE `attribution` ( But each time it fails, I have to uncompress that table and start the backup over. | |||||||||||||
| Comment by Vladislav Vaintroub [ 2021-05-11 ] | |||||||||||||
|
Looks like page validation (page_is_corrupted) bug, which is either marko or thiru | |||||||||||||
| Comment by Vladislav Vaintroub [ 2021-05-11 ] | |||||||||||||
|
Adding most probably cause | |||||||||||||
| Comment by Marko Mäkelä [ 2023-04-14 ] | |||||||||||||
|
Backups are supposed to be taken and prepared (logs applied) by the same major version of backup as the server. That said, there should not have been any compatibility-breaking changes other than | |||||||||||||
| Comment by Marko Mäkelä [ 2023-04-14 ] | |||||||||||||
|
Is this reproducible with newer versions of MariaDB? | |||||||||||||
| Comment by Will Mayall [ 2023-04-14 ] | |||||||||||||
|
I no longer work for the company where I experienced the issue. Sent from Yahoo Mail for iPhone On Friday, April 14, 2023, 6:55 AM, Marko Mäkelä (Jira) <jira@mariadb.org> wrote: Marko Mäkelä updated Is this reproducible with newer versions of MariaDB? – | |||||||||||||
| Comment by Marko Mäkelä [ 2023-04-16 ] | |||||||||||||
|
Thank you. In InnoDB itself, the fix of For mariadb-backup,
Already in MariaDB 10.3.21 the code looks different:
This looks definitely correct to me. I do not believe that this is a regression due to To analyze the bug, we would have needed a copy of the data file page that was claimed to be corrupted. |