Customer gets the following error when starting 10.6.17 immediately following sofware update from previous version, 10.6.16.
2024-06-19 21:05:44 10 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
|
2024-06-19 21:05:44 10 [ERROR] InnoDB: File './ibdata1' is corrupted
|
2024-06-19 21:05:44 10 [ERROR] InnoDB: We detected index corruption in an InnoDB type table. You have to dump + drop + reimport the table or, in a case of widespread corruption, dump all InnoDB tables and recreate the whole tablespace. If the mariadbd server crashes after the startup or when you dump the tables. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
|
2024-06-19 21:05:44 10 [ERROR] mariadbd: Index for table 'gtid_slave_pos' is corrupt; try to repair it
|
2024-06-19 21:05:44 10 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
|
2024-06-19 21:05:44 10 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
|
2024-06-19 21:05:44 0x7f266c669700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/storage/innobase/trx/trx0purge.cc line 268
|
For customer, this has occurred more than one time.
Customer discovered a workaround:
Start database one time with
Shutdown, remove innodb_force_recovery configuration.
Start database normally.
{"report":{"fcp":1300.7999999523163,"ttfb":502.59999990463257,"pageVisibility":"visible","entityId":129719,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"294d8df4-3fe5-441a-a522-d571d41878b0","navigationType":0,"readyForUser":1393.3999998569489,"redirectCount":0,"resourceLoadedEnd":1554.3999998569489,"resourceLoadedStart":507.5,"resourceTiming":[{"duration":314.2999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":507.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":507.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":821.7999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":314.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":507.69999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":507.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":822.1999998092651,"responseStart":0,"secureConnectionStart":0},{"duration":325.39999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":508,"connectEnd":508,"connectStart":508,"domainLookupEnd":508,"domainLookupStart":508,"fetchStart":508,"redirectEnd":0,"redirectStart":0,"requestStart":508,"responseEnd":833.3999998569489,"responseStart":833.3999998569489,"secureConnectionStart":508},{"duration":423.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":508.09999990463257,"connectEnd":508.09999990463257,"connectStart":508.09999990463257,"domainLookupEnd":508.09999990463257,"domainLookupStart":508.09999990463257,"fetchStart":508.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":508.09999990463257,"responseEnd":931.1999998092651,"responseStart":931.1999998092651,"secureConnectionStart":508.09999990463257},{"duration":426.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":508.2999999523163,"connectEnd":508.2999999523163,"connectStart":508.2999999523163,"domainLookupEnd":508.2999999523163,"domainLookupStart":508.2999999523163,"fetchStart":508.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":508.2999999523163,"responseEnd":935.0999999046326,"responseStart":935.0999999046326,"secureConnectionStart":508.2999999523163},{"duration":427,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":508.5,"connectEnd":508.5,"connectStart":508.5,"domainLookupEnd":508.5,"domainLookupStart":508.5,"fetchStart":508.5,"redirectEnd":0,"redirectStart":0,"requestStart":508.5,"responseEnd":935.5,"responseStart":935.5,"secureConnectionStart":508.5},{"duration":427.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":508.69999980926514,"connectEnd":508.69999980926514,"connectStart":508.69999980926514,"domainLookupEnd":508.69999980926514,"domainLookupStart":508.69999980926514,"fetchStart":508.69999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":508.69999980926514,"responseEnd":935.8999998569489,"responseStart":935.8999998569489,"secureConnectionStart":508.69999980926514},{"duration":475.60000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":508.89999985694885,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":508.89999985694885,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":984.5,"responseStart":0,"secureConnectionStart":0},{"duration":427.39999985694885,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":509,"connectEnd":509,"connectStart":509,"domainLookupEnd":509,"domainLookupStart":509,"fetchStart":509,"redirectEnd":0,"redirectStart":0,"requestStart":509,"responseEnd":936.3999998569489,"responseStart":936.3999998569489,"secureConnectionStart":509},{"duration":475.60000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":509.19999980926514,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":509.19999980926514,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":984.7999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":427.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":509.2999999523163,"connectEnd":509.2999999523163,"connectStart":509.2999999523163,"domainLookupEnd":509.2999999523163,"domainLookupStart":509.2999999523163,"fetchStart":509.2999999523163,"redirectEnd":0,"redirectStart":0,"requestStart":509.2999999523163,"responseEnd":936.8999998569489,"responseStart":936.8999998569489,"secureConnectionStart":509.2999999523163},{"duration":1038,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":516.0999999046326,"connectEnd":516.0999999046326,"connectStart":516.0999999046326,"domainLookupEnd":516.0999999046326,"domainLookupStart":516.0999999046326,"fetchStart":516.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":516.0999999046326,"responseEnd":1554.0999999046326,"responseStart":1554.0999999046326,"secureConnectionStart":516.0999999046326},{"duration":1033.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":520.6999998092651,"connectEnd":520.6999998092651,"connectStart":520.6999998092651,"domainLookupEnd":520.6999998092651,"domainLookupStart":520.6999998092651,"fetchStart":520.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":520.6999998092651,"responseEnd":1554.3999998569489,"responseStart":1554.3999998569489,"secureConnectionStart":520.6999998092651},{"duration":374,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":997.3999998569489,"connectEnd":997.3999998569489,"connectStart":997.3999998569489,"domainLookupEnd":997.3999998569489,"domainLookupStart":997.3999998569489,"fetchStart":997.3999998569489,"redirectEnd":0,"redirectStart":0,"requestStart":997.3999998569489,"responseEnd":1371.3999998569489,"responseStart":1371.3999998569489,"secureConnectionStart":997.3999998569489},{"duration":262.39999985694885,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1294,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1294,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1556.3999998569489,"responseStart":0,"secureConnectionStart":0},{"duration":218,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1439.5999999046326,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1439.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1657.5999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":173.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1440.5,"connectEnd":1440.5,"connectStart":1440.5,"domainLookupEnd":1440.5,"domainLookupStart":1440.5,"fetchStart":1440.5,"redirectEnd":0,"redirectStart":0,"requestStart":1440.5,"responseEnd":1613.5999999046326,"responseStart":1613.5999999046326,"secureConnectionStart":1440.5},{"duration":182.79999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":1441.0999999046326,"connectEnd":1441.0999999046326,"connectStart":1441.0999999046326,"domainLookupEnd":1441.0999999046326,"domainLookupStart":1441.0999999046326,"fetchStart":1441.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":1441.0999999046326,"responseEnd":1623.8999998569489,"responseStart":1623.8999998569489,"secureConnectionStart":1441.0999999046326}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":305,"responseStart":502,"responseEnd":520,"domLoading":506,"domInteractive":1608,"domContentLoadedEventStart":1608,"domContentLoadedEventEnd":1657,"domComplete":2040,"loadEventStart":2040,"loadEventEnd":2041,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1581},{"name":"bigPipe.sidebar-id.end","time":1581.7999999523163},{"name":"bigPipe.activity-panel-pipe-id.start","time":1581.7999999523163},{"name":"bigPipe.activity-panel-pipe-id.end","time":1584.6999998092651},{"name":"activityTabFullyLoaded","time":1713.1999998092651}],"measures":[],"correlationId":"684c1ead2373ed","effectiveType":"4g","downlink":9.2,"rtt":0,"serverDuration":132,"dbReadsTimeInMs":32,"dbConnsTimeInMs":42,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Major
70368744161280 is the byte offset of the largest 32-bit page number (4294967295, also known as FIL_NULL in the source code) when using the default innodb_page_size=16k.
In many persistent InnoDB data structures, such as linked lists of pages, FIL_NULL is being used as a "null pointer". A list somewhere is corrupted, and we do not know where.
It would be great if a copy of the corrupted data had been saved before engaging the workaround, and MariaDB 10.6.18 could be started on such a copy. I would like to see if the changes made in
MDEV-33325would have any impact on this. Ultimately, I would like to know where exactly the incorrect page number was being read without validation, so that we could get closer to the root cause (the write that corrupted the database).Based on the fact that innodb_force_recovery=1 seemed to heal the database, I assume that the corruption is somewhere in the undo logs, and as a result of this corruption, some purging of the committed transaction history was skipped. I would expect some CHECK TABLE…EXTENDED (
MDEV-24402) to report some orphan secondary index records. But, we must keep in mind that such orphans can also be caused by the known bug MDEV-29823 even if this undo log corruption bug wasn’t involved.