Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29511

"Page read from tablespace is corrupted" when starting mariadb service after crash (signal 6)

Details

    Description

      MariaDB throws a lot of "Page read from tablespace is corrupted" errors" when starting mariadb service after crash with signal 6.

      Crash:

      2022-09-12  1:50:08 0 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `mediawiki`.`searchindex`
      2022-09-12 01:50:08 0x7fb2f3fff700  InnoDB: Assertion failure in file ./storage/innobase/que/que0que.cc line 728
      InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mariadbd startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      220912  1:50:08 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.7.5-MariaDB-1:10.7.5+maria~deb11-log
      key_buffer_size=262144
      read_buffer_size=1048576
      max_used_connections=481
      max_threads=1002
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5156811 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x80000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x556d64f0082e]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x556d649eb8b5]
      

      Trying to start service again:

      2022-09-12  2:22:15 0 [Note] mariadbd: Aria engine: starting recovery
      recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 4 3 2 1 0
       (0.0 seconds); 
      2022-09-12  2:22:15 0 [Note] mariadbd: Aria engine: recovery done
      2022-09-12  2:22:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2022-09-12  2:22:15 0 [Note] InnoDB: Number of transaction pools: 1
      2022-09-12  2:22:15 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2022-09-12  2:22:15 0 [Note] InnoDB: Using liburing
      2022-09-12  2:22:15 0 [Note] InnoDB: Initializing buffer pool, total size = 34359738368, chunk size = 134217728
      2022-09-12  2:22:15 0 [Note] InnoDB: Completed initialization of buffer pool
      2022-09-12  2:22:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=12508540539949,12508639505793
      2022-09-12  2:22:30 0 [Note] InnoDB: Read redo log up to LSN=12509985608704
      2022-09-12  2:22:45 0 [Note] InnoDB: Read redo log up to LSN=12511455187968
      2022-09-12  2:22:50 0 [Note] InnoDB: Starting final batch to recover  56856 pages from redo log.
      2022-09-12  2:22:51 0 [ERROR] InnoDB: Not applying DELETE_ROW_FORMAT_DYNAMIC due to corruption on [page id: space=29347044, page number=575]
      2022-09-12  2:22:51 0 [ERROR] InnoDB: Set innodb_force_recovery=1 to ignore corruption.
      ... <- A lot of same error for different DB tables
      2022-09-12  2:22:51 0 [ERROR] InnoDB: Failed to read page 575 from file './hedera/userSession.ibd': Page read from tablespace is corrupted.
      2022-09-12  2:22:51 0 [ERROR] InnoDB: Failed to read page 7634 from file './vn/routeLog.ibd': Page read from tablespace is corrupted.
      ... <- A lot of same error for different DB tables
      2022-09-12  2:23:43 0 [ERROR] Plugin 'InnoDB' init function returned error.
      2022-09-12  2:23:43 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
      2022-09-12  2:23:43 0 [Note] Plugin 'FEEDBACK' is disabled.
      2022-09-12  2:23:43 0 [ERROR] Unknown/unsupported storage engine: InnoDB
      2022-09-12  2:23:43 0 [ERROR] Aborting
      

      Attachments

        Issue Links

          Activity

            juanferrer, a series of unscheduled releases including 10.7.6 took place this week because of the crash recovery regressions. The next scheduled releases will include 10.7.7.

            I believe that it should be safe to upgrade to 10.7.6. Since the time of that release, we have only found the recovery failure MDEV-29559, which unlike this bug is a "parsing" problem of a rare corner case, not a "write side" error.

            marko Marko Mäkelä added a comment - juanferrer , a series of unscheduled releases including 10.7.6 took place this week because of the crash recovery regressions. The next scheduled releases will include 10.7.7. I believe that it should be safe to upgrade to 10.7.6. Since the time of that release, we have only found the recovery failure MDEV-29559 , which unlike this bug is a "parsing" problem of a rare corner case, not a "write side" error.

            There are some good news from a 10.8.5 user in MDEV-29364.

            marko Marko Mäkelä added a comment - There are some good news from a 10.8.5 user in MDEV-29364 .
            juanferrer Juan Ferrer Toribio added a comment - - edited

            Hi marko, today I've upgraded our production MariaDB server from 10.7.4 to 10.7.6, after some time of testing I will give you feedback.

            juanferrer Juan Ferrer Toribio added a comment - - edited Hi marko , today I've upgraded our production MariaDB server from 10.7.4 to 10.7.6, after some time of testing I will give you feedback.

            juanferrer,
            How did it go with 10.7.6?

            elenst Elena Stepanova added a comment - juanferrer , How did it go with 10.7.6?
            juanferrer Juan Ferrer Toribio added a comment - - edited

            Hi elenst,

            There are still crashes in MariaDB 10.7.6, we had at least 3 (signal 11) since last upgrade, but the innodb recovery system worked fine in all of them, so I guess this particular issue is resolved.

            juanferrer Juan Ferrer Toribio added a comment - - edited Hi elenst , There are still crashes in MariaDB 10.7.6, we had at least 3 (signal 11) since last upgrade, but the innodb recovery system worked fine in all of them, so I guess this particular issue is resolved.

            People

              Unassigned Unassigned
              juanferrer Juan Ferrer Toribio
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.