It is observed that, while taking the backup using mariabackup(10.3.17) it's keep failing with following error:
Backup command used:
/var/lib/mysql/bin/mariabackup --defaults-file=/etc/my.cnf --user=**** --password=******* --backup --skip-encrypted-backup --compress --ftwrl-wait-timeout=5 --ftwrl-wait-threshold=300 --ftwrl-wait-query-type=all --target-dir=/tmp/backup/full_backup_2019
|
|
mariabackup output:
[00] 2019-11-16 06:29:24 Connecting to MySQL server host: localhost, user: xxx, password: xxx, port: 3306, socket: /var/lib/mysql/mysql.sock
|
[00] 2019-11-16 06:29:24 Using server version 10.3.17-MariaDB-log
|
/var/lib/mysql/bin/mariabackup based on MariaDB server 10.3.17-MariaDB Linux (x86_64)
|
....
|
....
|
[01] 2019-11-16 06:29:27 Compressing ./foo/bar.ibd to /tmp/backup/full_backup_2019/foo/bar.ibd.qp
|
[01] 2019-11-16 06:29:27 ...done
|
[01] 2019-11-16 06:29:27 Compressing ./foo/foobar.ibd to /tmp/backup/full_backup_2019/foo/foobar.ibd.qp
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:27 Database page corruption detected at page 5, retrying...
|
[01] 2019-11-16 06:29:28 Database page corruption detected at page 5, retrying...
|
[00] 2019-11-16 06:29:28 >> log scanned up to (30851334774)
|
[01] 2019-11-16 06:29:28 Error: failed to read page after 10 retries. File ./foo/foobar.ibd seems to be corrupted.
|
2019-11-16 6:29:28 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
|
.......
|
.......
|
InnoDB: End of page dump
|
2019-11-16 6:29:28 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 3454859770, calculated checksums for field1: crc32 3454859770, innodb 3654618756, page type 17855 == INDEX.none 3735928559, stored checksum in field2 3454859770, calculated checksums for field2: crc32 3454859770, innodb 4252287317, none 3735928559, page LSN 7 450897511, low 4 bytes of LSN at page end 450897511, page number (if stored to page already) 7, space id (if created with >= MySQL-4.1.1 and stored already) 1962
|
2019-11-16 6:29:28 0 [Note] InnoDB: Page may be an index page where index id is 5185
|
[01] 2019-11-16 06:29:28 mariabackup: xtrabackup_copy_datafile() failed.
|
[00] FATAL ERROR: 2019-11-16 06:29:28 failed to copy datafile.
|
|
Did tried, fixing the table (pointed as corrupted) with following, but the same issue is occurring.
set OLD_ALTER_TABLE=1
Alter table table_name engine=InnoDB
Alter table table_name FORCE
Take mysqldump of table and restore it to Database
{"report":{"fcp":1035.2999999523163,"ttfb":253.10000002384186,"pageVisibility":"visible","entityId":80598,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"3acdeceb-4b8a-4576-910d-8ec60793e70a","navigationType":0,"readyForUser":1137.7000000476837,"redirectCount":0,"resourceLoadedEnd":1166.3999999761581,"resourceLoadedStart":262,"resourceTiming":[{"duration":278.39999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":262,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":262,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":540.3999999761581,"responseStart":0,"secureConnectionStart":0},{"duration":278.40000009536743,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":262.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":262.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":540.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":287.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":262.39999997615814,"connectEnd":262.39999997615814,"connectStart":262.39999997615814,"domainLookupEnd":262.39999997615814,"domainLookupStart":262.39999997615814,"fetchStart":262.39999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":262.39999997615814,"responseEnd":549.7999999523163,"responseStart":549.7999999523163,"secureConnectionStart":262.39999997615814},{"duration":345.6999999284744,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":262.60000002384186,"connectEnd":262.60000002384186,"connectStart":262.60000002384186,"domainLookupEnd":262.60000002384186,"domainLookupStart":262.60000002384186,"fetchStart":262.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":262.60000002384186,"responseEnd":608.2999999523163,"responseStart":608.2999999523163,"secureConnectionStart":262.60000002384186},{"duration":349.1999999284744,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":262.7000000476837,"connectEnd":262.7000000476837,"connectStart":262.7000000476837,"domainLookupEnd":262.7000000476837,"domainLookupStart":262.7000000476837,"fetchStart":262.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":262.7000000476837,"responseEnd":611.8999999761581,"responseStart":611.8999999761581,"secureConnectionStart":262.7000000476837},{"duration":349.2999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":263,"connectEnd":263,"connectStart":263,"domainLookupEnd":263,"domainLookupStart":263,"fetchStart":263,"redirectEnd":0,"redirectStart":0,"requestStart":263,"responseEnd":612.2999999523163,"responseStart":612.2999999523163,"secureConnectionStart":263},{"duration":349.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":263.10000002384186,"connectEnd":263.10000002384186,"connectStart":263.10000002384186,"domainLookupEnd":263.10000002384186,"domainLookupStart":263.10000002384186,"fetchStart":263.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":263.10000002384186,"responseEnd":612.6000000238419,"responseStart":612.6000000238419,"secureConnectionStart":263.10000002384186},{"duration":423.8000000715256,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":263.2999999523163,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":263.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":687.1000000238419,"responseStart":0,"secureConnectionStart":0},{"duration":349.5,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":263.39999997615814,"connectEnd":263.39999997615814,"connectStart":263.39999997615814,"domainLookupEnd":263.39999997615814,"domainLookupStart":263.39999997615814,"fetchStart":263.39999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":263.39999997615814,"responseEnd":612.8999999761581,"responseStart":612.8999999761581,"secureConnectionStart":263.39999997615814},{"duration":423.7000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":263.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":263.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":687.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":349.89999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":263.7000000476837,"connectEnd":263.7000000476837,"connectStart":263.7000000476837,"domainLookupEnd":263.7000000476837,"domainLookupStart":263.7000000476837,"fetchStart":263.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":263.7000000476837,"responseEnd":613.6000000238419,"responseStart":613.6000000238419,"secureConnectionStart":263.7000000476837},{"duration":511.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":264.89999997615814,"connectEnd":264.89999997615814,"connectStart":264.89999997615814,"domainLookupEnd":264.89999997615814,"domainLookupStart":264.89999997615814,"fetchStart":264.89999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":264.89999997615814,"responseEnd":776.2999999523163,"responseStart":776.2999999523163,"secureConnectionStart":264.89999997615814},{"duration":901.3999999761581,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":265,"connectEnd":265,"connectStart":265,"domainLookupEnd":265,"domainLookupStart":265,"fetchStart":265,"redirectEnd":0,"redirectStart":0,"requestStart":265,"responseEnd":1166.3999999761581,"responseStart":1166.3999999761581,"secureConnectionStart":265},{"duration":80.79999995231628,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":698.2000000476837,"connectEnd":698.2000000476837,"connectStart":698.2000000476837,"domainLookupEnd":698.2000000476837,"domainLookupStart":698.2000000476837,"fetchStart":698.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":698.2000000476837,"responseEnd":779,"responseStart":779,"secureConnectionStart":698.2000000476837}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":8,"responseStart":253,"responseEnd":259,"domLoading":259,"domInteractive":1209,"domContentLoadedEventStart":1209,"domContentLoadedEventEnd":1253,"domComplete":1548,"loadEventStart":1548,"loadEventEnd":1549,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1189},{"name":"bigPipe.sidebar-id.end","time":1189.6000000238419},{"name":"bigPipe.activity-panel-pipe-id.start","time":1189.7999999523163},{"name":"bigPipe.activity-panel-pipe-id.end","time":1191.6000000238419},{"name":"activityTabFullyLoaded","time":1274.3999999761581}],"measures":[],"correlationId":"4280caf4726d1b","effectiveType":"4g","downlink":9.3,"rtt":0,"serverDuration":162,"dbReadsTimeInMs":29,"dbConnsTimeInMs":40,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
marko Yes, the "corruption" can be detected by innochecksum with
MDEV-19871fix. And yes, we can zero out non-allocated pages both with innochecksum and "mariabackup --prepare". But I have the following concerns:1. innochecksum is the tool which is supposed to check tablespaces and does not modify them. We have option to rewrite checksum algorithm. But it modifies only checksums. Should we turn the tool for checksums check to the tool for corruptions fixing?
2. "mariabackup --prepare" could fix such pages. But, as I understood, there must not be non-zero non-allocated pages. And we need to understand what is the source of such pages. And when we understand it, the code to fix such pages will be useless. There will be one more option which is rarely used, and which we have to maintain until some major version.