Type:
Bug
Priority:
Critical
Resolution:
Fixed
Affects Version/s:
10.6
mleich provided a copy of a data directory as well as rr replay traces that leads to a recovery failure with an assertion failure.
Unfortunately, the data directory enables encryption and includes a 96 MiB ib_logfile0 that wrapped around once. Because encrypted data does not compress well, a compressed copy of the data directory would be too large to attach here.
The rr replay trace from before the crash is of limited use, because rr replay would report a replay divergence near the end of the trace. Right before the 10.6 version of MDEV-29911 , recovery will appear to succeed:
10.6 1fe830b56a2bd9b12b643d7b39417255215ae5da
2023-05-29 16:45:43 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142804091,143054310
2023-05-29 16:45:43 0 [Note] InnoDB: Starting final batch to recover 174 pages from redo log.
2023-05-29 16:45:43 0 [Note] InnoDB: Last binlog file './mysql-bin.000001', position 2691230
2023-05-29 16:45:43 0 [Note] InnoDB: Removing encryption and resizing redo log from 100663296 to 4294967296 bytes; LSN=151202977
If I attempt recovery with the fix of MDEV-29911 , I will hit a debug assertion instead:
10.6 f2c17cc9d9bcd634887846d3064bcb71243f9cc0
2023-05-29 16:47:48 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=142804091,143054310
mariadbd: /mariadb/10.6/storage/innobase/log/log0recv.cc:3629: void recv_sys_t::apply(bool): Assertion `!last_batch || recovered_lsn == scanned_lsn' failed.
During the time of the assertion failure, we have recv_sys.recovered_lsn==151202728 and recv_sys.scanned_lsn==151209984 . The largest observed *contiguous_lsn in recv_scan_log_recs() is 150516736 .
In the non-crashing run, recovery only proceeded up to 151202977 . This would seem to suggest that the "successful" run without MDEV-29911 may be incorrect.
MariaDB Server 10.8 and later versions could be unaffected by this exact bug, because the separate log block and log record parsers were unified when MDEV-14425 replaced the 512-byte log blocks with mini-transaction-sized log blocks. Other recovery bugs are possible; in MDEV-31353 there is a recent example.
relates to
MDEV-29911
InnoDB recovery and mariadb-backup --prepare fail to report detailed progress
Closed
Marko Mäkelä
made changes -
2023-05-29 14:13
Status
Open
[ 1
]
In Progress
[ 3
]
Marko Mäkelä
made changes -
2023-05-30 14:53
issue.field.resolutiondate
2023-05-30 14:53:30.0
2023-05-30 14:53:30.108
Marko Mäkelä
made changes -
2023-05-30 14:53
Fix Version/s
10.6.14
[ 28914
]
Fix Version/s
10.6
[ 24028
]
Resolution
Fixed
[ 1
]
Status
In Progress
[ 3
]
Closed
[ 6
]
Marko Mäkelä
made changes -
2023-05-30 14:53
Labels
corruption recovery rr-profile
corruption debug recovery rr-profile-analyzed
Marko Mäkelä
made changes -
2023-05-30 14:53
Labels
corruption debug recovery rr-profile-analyzed
debug recovery rr-profile-analyzed
Marko Mäkelä
made changes -
2023-06-07 08:40
Fix Version/s
10.6.15
[ 29013
]
Fix Version/s
10.6.14
[ 28914
]
Sergei Golubchik
made changes -
2023-06-07 15:19
Fix Version/s
10.9.8
[ 29015
]
Fix Version/s
10.10.6
[ 29017
]
Fix Version/s
10.11.5
[ 29019
]
Fix Version/s
11.0.3
[ 28920
]
Marko Mäkelä
made changes -
2023-06-07 15:28
Fix Version/s
11.0.3
[ 28920
]
Fix Version/s
10.9.8
[ 29015
]
Fix Version/s
10.10.6
[ 29017
]
Fix Version/s
10.11.5
[ 29019
]
{"report":{"fcp":1369.3999996185303,"ttfb":432.0999994277954,"pageVisibility":"visible","entityId":122345,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"e7f7011c-3c1c-480a-ba5f-7393f936fb82","navigationType":0,"readyForUser":1483.5999994277954,"redirectCount":0,"resourceLoadedEnd":1623,"resourceLoadedStart":438.5,"resourceTiming":[{"duration":374.0999994277954,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":438.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":438.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":812.5999994277954,"responseStart":0,"secureConnectionStart":0},{"duration":375.8999996185303,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":438.80000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":438.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":814.6999998092651,"responseStart":0,"secureConnectionStart":0},{"duration":430.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":439,"connectEnd":439,"connectStart":439,"domainLookupEnd":439,"domainLookupStart":439,"fetchStart":439,"redirectEnd":0,"redirectStart":0,"requestStart":439,"responseEnd":869.8000001907349,"responseStart":869.8000001907349,"secureConnectionStart":439},{"duration":540.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/2bf333562ca6724060a9d5f1535471f6/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":439.0999994277954,"connectEnd":439.0999994277954,"connectStart":439.0999994277954,"domainLookupEnd":439.0999994277954,"domainLookupStart":439.0999994277954,"fetchStart":439.0999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":439.0999994277954,"responseEnd":979.1999998092651,"responseStart":979.1999998092651,"secureConnectionStart":439.0999994277954},{"duration":544.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":439.3999996185303,"connectEnd":439.3999996185303,"connectStart":439.3999996185303,"domainLookupEnd":439.3999996185303,"domainLookupStart":439.3999996185303,"fetchStart":439.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":439.3999996185303,"responseEnd":984.3000001907349,"responseStart":984.3000001907349,"secureConnectionStart":439.3999996185303},{"duration":547.2000007629395,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":439.5999994277954,"connectEnd":439.5999994277954,"connectStart":439.5999994277954,"domainLookupEnd":439.5999994277954,"domainLookupStart":439.5999994277954,"fetchStart":439.5999994277954,"redirectEnd":0,"redirectStart":0,"requestStart":439.5999994277954,"responseEnd":986.8000001907349,"responseStart":986.8000001907349,"secureConnectionStart":439.5999994277954},{"duration":547.5999994277954,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":439.80000019073486,"connectEnd":439.80000019073486,"connectStart":439.80000019073486,"domainLookupEnd":439.80000019073486,"domainLookupStart":439.80000019073486,"fetchStart":439.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":439.80000019073486,"responseEnd":987.3999996185303,"responseStart":987.3999996185303,"secureConnectionStart":439.80000019073486},{"duration":579.8000001907349,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":440,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":440,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1019.8000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":548.1000003814697,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":440.19999980926514,"connectEnd":440.19999980926514,"connectStart":440.19999980926514,"domainLookupEnd":440.19999980926514,"domainLookupStart":440.19999980926514,"fetchStart":440.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":440.19999980926514,"responseEnd":988.3000001907349,"responseStart":988.3000001907349,"secureConnectionStart":440.19999980926514},{"duration":579.6999998092651,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":440.30000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":440.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1020,"responseStart":0,"secureConnectionStart":0},{"duration":548.9000005722046,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":440.3999996185303,"connectEnd":440.3999996185303,"connectStart":440.3999996185303,"domainLookupEnd":440.3999996185303,"domainLookupStart":440.3999996185303,"fetchStart":440.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":440.3999996185303,"responseEnd":989.3000001907349,"responseStart":989.1999998092651,"secureConnectionStart":440.3999996185303},{"duration":737.7999992370605,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":447.30000019073486,"connectEnd":447.30000019073486,"connectStart":447.30000019073486,"domainLookupEnd":447.30000019073486,"domainLookupStart":447.30000019073486,"fetchStart":447.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":447.30000019073486,"responseEnd":1185.0999994277954,"responseStart":1185.0999994277954,"secureConnectionStart":447.30000019073486},{"duration":1164.1999998092651,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":454.3999996185303,"connectEnd":454.3999996185303,"connectStart":454.3999996185303,"domainLookupEnd":454.3999996185303,"domainLookupStart":454.3999996185303,"fetchStart":454.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":454.3999996185303,"responseEnd":1618.5999994277954,"responseStart":1618.5999994277954,"secureConnectionStart":454.3999996185303},{"duration":153.60000038146973,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1037.3999996185303,"connectEnd":1037.3999996185303,"connectStart":1037.3999996185303,"domainLookupEnd":1037.3999996185303,"domainLookupStart":1037.3999996185303,"fetchStart":1037.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":1037.3999996185303,"responseEnd":1191,"responseStart":1191,"secureConnectionStart":1037.3999996185303},{"duration":302.9000005722046,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":1318.3999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1318.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1621.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":300.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":1319.3000001907349,"connectEnd":1319.3000001907349,"connectStart":1319.3000001907349,"domainLookupEnd":1319.3000001907349,"domainLookupStart":1319.3000001907349,"fetchStart":1319.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":1319.3000001907349,"responseEnd":1619.8000001907349,"responseStart":1619.8000001907349,"secureConnectionStart":1319.3000001907349},{"duration":286.0999994277954,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1360.3000001907349,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1360.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1646.3999996185303,"responseStart":0,"secureConnectionStart":0},{"duration":303.19999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":1319.8000001907349,"connectEnd":1319.8000001907349,"connectStart":1319.8000001907349,"domainLookupEnd":1319.8000001907349,"domainLookupStart":1319.8000001907349,"fetchStart":1319.8000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":1319.8000001907349,"responseEnd":1623,"responseStart":1623,"secureConnectionStart":1319.8000001907349}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":238,"responseStart":432,"responseEnd":455,"domLoading":436,"domInteractive":1655,"domContentLoadedEventStart":1655,"domContentLoadedEventEnd":1702,"domComplete":2037,"loadEventStart":2037,"loadEventEnd":2037,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1627.8999996185303},{"name":"bigPipe.sidebar-id.end","time":1628.8999996185303},{"name":"bigPipe.activity-panel-pipe-id.start","time":1629},{"name":"bigPipe.activity-panel-pipe-id.end","time":1632},{"name":"activityTabFullyLoaded","time":1725.8999996185303}],"measures":[],"correlationId":"998a39bd70223d","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":132,"dbReadsTimeInMs":25,"dbConnsTimeInMs":34,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}