Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33922

InnoDB undo log tablespace file corruption

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.11.7, 10.11.8
    • N/A
    • Storage Engine - InnoDB
    • CloudLinux release 7.9 (Boris Yegorov)
      CloudLinux release 8.9 (Anatoly Levchenko)

    Description

      Hello, we've had 15 separate MariaDB instances in the past 4 months run into an issue where the MariaDB server suddenly crashes, with an indication of a corrupted undo tablespace file, and starts to crash loop until it's begun with innodb_force_recovery=1 to ignore the corrupted page, as is described in the error logs. All of the servers are using `innodb_undo_tablespaces=3`in `my.cnf`.

      We can't find a reliable way to reproduce the issue. We initially thought it was brought over during 10.6 -> 10.11 upgrades, but the most recent case is a month-old server that started with 10.11.7. If there's anything else you need to help debug the issue, we'll be happy to provide it.

      Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16  7:44:58 1447973524 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: .//undo003
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16  7:44:58 1447973524 [ERROR] InnoDB: File './/undo003' is corrupted
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 07:44:58 0x7f7f9013f700  InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB1011-10.11.7/mariadb-10.11.7/storage/innobase/trx/trx0purge.cc line 268
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Failing assertion: undo_page
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: We intentionally generate a memory trap.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: If you get repeated assertion failures or crashes, even
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: immediately after the mariadbd startup, there may be
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: about forcing recovery.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: 240416  7:44:58 [ERROR] mysqld got signal 6 ;
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Sorry, we probably made a mistake, and this is a bug.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Your assistance in bug reporting will enable us to fix this for the next release.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: We will try our best to scrape up some info that will hopefully help
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: diagnose the problem, but since we have already crashed,
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: something is definitely wrong and this may fail.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Server version: 10.11.7-MariaDB-cll-lve source revision: 87e13722a95af5d9378d990caf48cb6874439347
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: key_buffer_size=2147483648
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: read_buffer_size=4194304
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: max_used_connections=1205
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: max_threads=2002
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: thread_count=320
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: It is possible that mysqld could use up to
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18551172 K  bytes of memory
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Hope that's ok; if not, decrease some variables in the equation.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Thread pointer: 0x7f81165dbc58
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: Attempting backtrace. You can use the following information to find out
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: where mysqld died. If you see no messages after this, something went
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: terribly wrong...
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: stack_bottom = 0x7f7f9013d358 thread_stack 0x49000
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 07:44:58 0x7f7a770c0700  InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB1011-10.11.7/mariadb-10.11.7/storage/innobase/buf/buf0lru.cc line 281
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Failing assertion: !block->page.in_file()
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: We intentionally generate a memory trap.
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: If you get repeated assertion failures or crashes, even
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: immediately after the mariadbd startup, there may be
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: corruption in the InnoDB tablespace. Please refer to
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      		 
      Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: about forcing recovery.
      		 
      Apr 16 07:45:05 host1339 systemd-coredump[2183663]: Process 173461 (mysqld) of user 991 dumped core.
      		 
      ...
      		 
      Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16  8:02:23 0 [ERROR] InnoDB: Corrupted page identifier at 3543005337948; set innodb_force_recovery=1 to ignore the record.
      		 
      Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16  8:02:23 0 [Note] InnoDB: End of log at LSN=3543005337948
      		 
      Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16  8:02:23 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33189 Server crash after reading outside of bounds on ibdata1 , file corrupted, no auto-recovery

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34233 InnoDB crashes due to corrupted ibdata1 (Assertion failure in innodb.undo_page)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-34453 Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: ./ibdata1

          • Major - Major loss of function.
          • Closed

          Activity

            Transition Time In Source Status Execution Times
            Marko Mäkelä made transition -
            Open Needs Feedback
            		222d 23h 55m 1
            Julien Fritsch made transition -
            Needs Feedback Closed
            		34d 1h 10m 1

            People

              Unassigned Unassigned
              arnklo Arnas Klova
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.