Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33189

Server crash after reading outside of bounds on ibdata1 , file corrupted, no auto-recovery



    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 11.2, 11.4, 10.11.6
    • N/A
    • None
    • Debian GNU/Linux 11 (bullseye)
      Dell PowerEdge R750
      XFS filesystem


      We experienced a one-time server crash in production, so far not reproducible.

      We are running MariaDB 10.11.6 (1:10.11.6+maria~deb11) installed from a MariaDB repo mirror on Debian GNU/Linux 11 (bullseye) as the database primary for a read- and write-heavy application. It runs on a bare-metal server Dell PowerEdge R750 with 64 cores (Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz), 512 GiB RAM on a software RAID-1 NVMe with an XFS filesystem.

      The server crash happened on 2023-12-21. On 2023-12-02 we had upgraded to v10.11.6. Prior to that the DB ran without any problems on v10.6.7 for almost 1.5 years.

      The server itself crashed with:

      InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1
      InnoDB: File './ibdata1' is corrupted

      and two assertion failures in trx0undo.cc and buf0lru.cc. All subsequent restart attempts failed so we switched the application over to the replica database.

      We did not attempt any forced recovery. The assertion failures:

      InnoDB: Assertion failure in file ./storage/innobase/trx/trx0undo.cc line 1416
      InnoDB: Failing assertion: rollback
      231221 14:24:48 [ERROR] mysqld got signal 6 ;

      The backtrace only gave one line before having the next assertion failure.

      stack_bottom = 0x7f614d088cd8 thread_stack 0x49000
      InnoDB: Assertion failure in file ./storage/innobase/buf/buf0lru.cc line 285
      InnoDB: Failing assertion: !block->page.in_file()

      See attachment db-syslog.2023-12-21.txt for all the relevant syslog entries.

      We have preserved the corrupt 716 MiB ibdata1 (750780416 B) file for further inspection, should the need arise.


        Issue Links



              marko Marko Mäkelä
              wschemmel Wolfgang Schemmel
              0 Vote for this issue
              5 Start watching this issue



                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.