We experienced a server crash, on a production server.
This is the second time it happens on the same box. The previous time was reported in MDEV-33189
(Hardware unchanged:
Dell PowerEdge R750 with 64 cores (Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz),
512 GiB RAM
software RAID-1 NVMe)
Linux db01 5.10.0-32-amd64 #1 SMP Debian 5.10.223-1 (2024-08-10) x86_64 GNU/Linux
mariadb-server 1:10.11.7+maria~deb11
[ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
|
[ERROR] InnoDB: File './ibdata1' is corrupted
|
[ERROR] InnoDB: We detected index corruption in an InnoDB type table
|
: We will try our best to scrape up some info that will hopefully help
|
: diagnose the problem, but since we have already crashed,
|
: something is definitely wrong and this may fail.
|
: Server version: 10.11.7-MariaDB-1:10.11.7+maria~deb11-log source revision: 87e13722a95af5d9378d990caf48cb6874439347
|
: key_buffer_size=134217728
|
: read_buffer_size=131072
|
: max_used_connections=5
|
: max_threads=4002
|
: thread_count=7
|
: It is possible that mysqld could use up to
|
: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4848246 K bytes of memory
|
: Hope that's ok; if not, decrease some variables in the equation.
|
: Thread pointer: 0x7f5d30001598
|
: Attempting backtrace. You can use the following information to find out
|
: where mysqld died. If you see no messages after this, something went
|
: terribly wrong...
|
: stack_bottom = 0x7f5d6d372660 thread_stack 0x49000
|
: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x562e41d0632e]
|
: /usr/sbin/mariadbd(handle_fatal_signal+0x475)[0x562e417df465]
|
: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f68ea15a140]
|
: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f68e9c92d51]
|
: /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f68e9c7c537]
|
: /usr/sbin/mariadbd(+0x6ab7d9)[0x562e414087d9]
|
: /usr/sbin/mariadbd(+0x6a71d9)[0x562e414041d9]
|
: /usr/sbin/mariadbd(+0xe6a9e1)[0x562e41bc79e1]
|
: /usr/sbin/mariadbd(+0x70b580)[0x562e41468580]
|
: /usr/sbin/mariadbd(+0xe68be0)[0x562e41bc5be0]
|
: /usr/sbin/mariadbd(+0xe68c39)[0x562e41bc5c39]
|
: /usr/sbin/mariadbd(+0xe5aead)[0x562e41bb7ead]
|
: /usr/sbin/mariadbd(+0xe59b11)[0x562e41bb6b11]
|
: /usr/sbin/mariadbd(+0xd5293e)[0x562e41aaf93e]
|
: /usr/sbin/mariadbd(_Z17ha_rollback_transP3THDb+0x120)[0x562e417e3550]
|
: /usr/sbin/mariadbd(_Z14trans_rollbackP3THD+0x43)[0x562e416c0473]
|
: /usr/sbin/mariadbd(_ZN14rpl_group_info15cleanup_contextEP3THDb+0x70)[0x562e416a58c0]
|
: /usr/sbin/mariadbd(handle_slave_sql+0xe1b)[0x562e414c26eb]
|
: /usr/sbin/mariadbd(+0xca9cc2)[0x562e41a06cc2]
|
And then:
|
2024-11-11 01:18:53 0x7f5d6d2c3700 InnoDB: Assertion failure in file ./storage/innobase/buf/buf0lru.cc line 281
|
mariadbd[3792424]: InnoDB: Failing assertion: !block->page.in_file()
|
mariadbd[3792424]: InnoDB: We intentionally generate a memory trap.
|
mariadbd[3792424]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
mariadbd[3792424]: InnoDB: If you get repeated assertion failures or crashes, even
|
mariadbd[3792424]: InnoDB: immediately after the mariadbd startup, there may be
|
mariadbd[3792424]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
mariadbd[3792424]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
mariadbd[3792424]: InnoDB: about forcing recovery.
|
We have kept the original ibdata1 ibtmp1 and ib_logfile0, as well as the database directory, in case thay are needed for debugging purposes.
Forced recovery flags were not used. We restarted the Server operations by restoring from backup.
{"report":{"fcp":953.7000000476837,"ttfb":207.70000004768372,"pageVisibility":"visible","entityId":131552,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"ed8d3fed-402e-42cc-a13b-8964cd25e769","navigationType":0,"readyForUser":1022.2000000476837,"redirectCount":0,"resourceLoadedEnd":1158.4000000953674,"resourceLoadedStart":213.29999995231628,"resourceTiming":[{"duration":298.7999999523163,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":213.29999995231628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":213.29999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":512.0999999046326,"responseStart":0,"secureConnectionStart":0},{"duration":298.60000014305115,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":213.59999990463257,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":213.59999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":512.2000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":308.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":213.70000004768372,"connectEnd":213.70000004768372,"connectStart":213.70000004768372,"domainLookupEnd":213.70000004768372,"domainLookupStart":213.70000004768372,"fetchStart":213.70000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":213.70000004768372,"responseEnd":522.4000000953674,"responseStart":522.4000000953674,"secureConnectionStart":213.70000004768372},{"duration":338.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":213.79999995231628,"connectEnd":213.79999995231628,"connectStart":213.79999995231628,"domainLookupEnd":213.79999995231628,"domainLookupStart":213.79999995231628,"fetchStart":213.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":213.79999995231628,"responseEnd":552.2000000476837,"responseStart":552.2000000476837,"secureConnectionStart":213.79999995231628},{"duration":334.7000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":213.79999995231628,"connectEnd":213.79999995231628,"connectStart":213.79999995231628,"domainLookupEnd":213.79999995231628,"domainLookupStart":213.79999995231628,"fetchStart":213.79999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":213.79999995231628,"responseEnd":548.5,"responseStart":548.5,"secureConnectionStart":213.79999995231628},{"duration":338.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":214,"connectEnd":214,"connectStart":214,"domainLookupEnd":214,"domainLookupStart":214,"fetchStart":214,"redirectEnd":0,"redirectStart":0,"requestStart":214,"responseEnd":552.5999999046326,"responseStart":552.5999999046326,"secureConnectionStart":214},{"duration":339,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":214.09999990463257,"connectEnd":214.09999990463257,"connectStart":214.09999990463257,"domainLookupEnd":214.09999990463257,"domainLookupStart":214.09999990463257,"fetchStart":214.09999990463257,"redirectEnd":0,"redirectStart":0,"requestStart":214.09999990463257,"responseEnd":553.0999999046326,"responseStart":553.0999999046326,"secureConnectionStart":214.09999990463257},{"duration":339.2000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":214.20000004768372,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.20000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":553.4000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":339.2000000476837,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":214.20000004768372,"connectEnd":214.20000004768372,"connectStart":214.20000004768372,"domainLookupEnd":214.20000004768372,"domainLookupStart":214.20000004768372,"fetchStart":214.20000004768372,"redirectEnd":0,"redirectStart":0,"requestStart":214.20000004768372,"responseEnd":553.4000000953674,"responseStart":553.4000000953674,"secureConnectionStart":214.20000004768372},{"duration":339.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":214.29999995231628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":214.29999995231628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":553.7999999523163,"responseStart":0,"secureConnectionStart":0},{"duration":339.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":214.5,"connectEnd":214.5,"connectStart":214.5,"domainLookupEnd":214.5,"domainLookupStart":214.5,"fetchStart":214.5,"redirectEnd":0,"redirectStart":0,"requestStart":214.5,"responseEnd":554,"responseStart":554,"secureConnectionStart":214.5},{"duration":550.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":219.90000009536743,"connectEnd":219.90000009536743,"connectStart":219.90000009536743,"domainLookupEnd":219.90000009536743,"domainLookupStart":219.90000009536743,"fetchStart":219.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":219.90000009536743,"responseEnd":770.5,"responseStart":770.5,"secureConnectionStart":219.90000009536743},{"duration":938.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":219.90000009536743,"connectEnd":219.90000009536743,"connectStart":219.90000009536743,"domainLookupEnd":219.90000009536743,"domainLookupStart":219.90000009536743,"fetchStart":219.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":219.90000009536743,"responseEnd":1158.4000000953674,"responseStart":1158.4000000953674,"secureConnectionStart":219.90000009536743},{"duration":402.7999999523163,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":754.2000000476837,"connectEnd":754.2000000476837,"connectStart":754.2000000476837,"domainLookupEnd":754.2000000476837,"domainLookupStart":754.2000000476837,"fetchStart":754.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":754.2000000476837,"responseEnd":1157,"responseStart":1156.9000000953674,"secureConnectionStart":754.2000000476837},{"duration":227.29999995231628,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":932.4000000953674,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":932.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1159.7000000476837,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":67,"responseStart":208,"responseEnd":213,"domLoading":211,"domInteractive":1183,"domContentLoadedEventStart":1183,"domContentLoadedEventEnd":1219,"domComplete":1859,"loadEventStart":1859,"loadEventEnd":1860,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1163.5999999046326},{"name":"bigPipe.sidebar-id.end","time":1164.4000000953674},{"name":"bigPipe.activity-panel-pipe-id.start","time":1164.5999999046326},{"name":"bigPipe.activity-panel-pipe-id.end","time":1167},{"name":"activityTabFullyLoaded","time":1226}],"measures":[],"correlationId":"7caf21f79de28e","effectiveType":"4g","downlink":9.4,"rtt":0,"serverDuration":79,"dbReadsTimeInMs":13,"dbConnsTimeInMs":20,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Bug
Critical
