Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26537

InnoDB corrupts files due to incorrect st_blksize calculation

Details

    Description

      Hey,

      I have a job that migrates an old database to in a different format (my custom, php based forum migrating to a django-based one).

      Recently, I've seen the mariadb server crashing randomly, leaving other databases and their tables corrupted. I can't just pinpoint out the exact issue, but I can provide the traceback and coredump, in hopes of you finding and fixing the bug ASAP.

      This posted crashlog is from executing mariadb-check -c --all-databases -S /tmp/mysql.sock, that according to its output, crashes on a certain CHECK TABLE command.

      The crash log from mysqld.err:
      ===
      2021-09-04 11:15:41 0x80f1b5d00 InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb105-server/work/mariadb-10.5.
      12/storage/innobase/btr/btr0btr.cc line 4763
      InnoDB: Failing assertion: !page_is_empty(page) || (level == 0 && page_get_page_no(page) == dict_index_get_page(index))
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
      InnoDB: about forcing recovery.
      210904 11:15:41 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

      Server version: 10.5.12-MariaDB
      key_buffer_size=16777216
      read_buffer_size=8388608
      max_used_connections=35
      max_threads=153
      thread_count=34
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11300560 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x81e206b98
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fffdafdef38 thread_stack 0x49000
      0x130cd7c <my_print_stacktrace+0x3c> at /usr/local/libexec/mariadbd
      0xc6e6af <handle_fatal_signal+0x28f> at /usr/local/libexec/mariadbd
      0x801904e00 <_pthread_sigmask+0x530> at /lib/libthr.so.3

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x81e24def0): CHECK TABLE `forum_event_eventresponse`

      Connection ID (thread ID): 138
      Status: NOT_KILLED

      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash.
      Core pattern: %N.core
      ===

      Since I couldn't upload the core file here (max filesize exceeded), please download the gzipped coredump from here: https://wetransfer.com/downloads/58bcaac03b364010532ebddd214c7dbd20210904093632/a91858186386ac9749129b6f2632db3520210904093659/8c9aa0

      Attachments

        Issue Links

          Activity

            A source tree origin/bb-10.5-marko commit 716f04a40f225af11fa88e793441da96dd0ddf21
            containing a fix for the current MDEV-26537 behaved well in RQG testing (innodb_page_size=4K only).
            

            mleich Matthias Leich added a comment - A source tree origin/bb-10.5-marko commit 716f04a40f225af11fa88e793441da96dd0ddf21 containing a fix for the current MDEV-26537 behaved well in RQG testing (innodb_page_size=4K only).

            Will this fix only be applied to 10.5 tree or will it be also applied on the other versions?

            falsovsky Pedro de Oliveira added a comment - Will this fix only be applied to 10.5 tree or will it be also applied on the other versions?
            marko Marko Mäkelä added a comment - I have pushed the fix to the 10.2 branch and merged up to 10.5 so far.
            serg Sergei Golubchik added a comment - - edited

            It was fixed in FreeBSD Port collection on September 12. New, fixed, versions:

            • 10.3.31_1
            • 10.4.21_1
            • 10.5.12_1
            serg Sergei Golubchik added a comment - - edited It was fixed in FreeBSD Port collection on September 12 . New, fixed, versions: 10.3.31_1 10.4.21_1 10.5.12_1
            marko Marko Mäkelä added a comment - - edited

            It turns out that the fix causes a debug assertion failure on crash recovery (MDEV-27016) when using ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=2 or 1, at least on 10.5 and 10.6. I think that the scenario must involve DDL, because I was unable to repeat it with DML (the file size would always jump from 64 KiB to 5 MiB).

            diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/buf0flu.cc
            index 583b793aff7..d3b6b1f9371 100644
            --- a/storage/innobase/buf/buf0flu.cc
            +++ b/storage/innobase/buf/buf0flu.cc
            @@ -962,7 +962,10 @@ static page_id_t buf_flush_check_neighbors(const fil_space_t &space,
                                                        page_id_t &id, bool contiguous,
                                                        bool lru)
             {
            -  ut_ad(id.page_no() < space.size);
            +  ut_ad(id.page_no() < space.size_in_header);
            +  ut_ad(id.page_no() < space.size +
            +        (space.physical_size() == 2048 ? 1
            +         : space.physical_size() == 1024 ? 3 : 0));
               /* When flushed, dirty blocks are searched in neighborhoods of this
               size, and flushed along with the original page. */
               const ulint s= buf_pool.curr_size / 16;
            

            Both space.size and space.committed_size would be 64, while space.size_in_header would be 65 for the copy of the data directory. The space.size is based on the file size, and we’d never extend files in increments smaller than 4096 bytes.

            This patch has not gone through our stress tests yet. In any case, only cmake -DCMAKE_BUILD_TYPE=Debug should be affected by this.

            marko Marko Mäkelä added a comment - - edited It turns out that the fix causes a debug assertion failure on crash recovery ( MDEV-27016 ) when using ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=2 or 1, at least on 10.5 and 10.6. I think that the scenario must involve DDL, because I was unable to repeat it with DML (the file size would always jump from 64 KiB to 5 MiB). diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/buf0flu.cc index 583b793aff7..d3b6b1f9371 100644 --- a/storage/innobase/buf/buf0flu.cc +++ b/storage/innobase/buf/buf0flu.cc @@ -962,7 +962,10 @@ static page_id_t buf_flush_check_neighbors(const fil_space_t &space, page_id_t &id, bool contiguous, bool lru) { - ut_ad(id.page_no() < space.size); + ut_ad(id.page_no() < space.size_in_header); + ut_ad(id.page_no() < space.size + + (space.physical_size() == 2048 ? 1 + : space.physical_size() == 1024 ? 3 : 0)); /* When flushed, dirty blocks are searched in neighborhoods of this size, and flushed along with the original page. */ const ulint s= buf_pool.curr_size / 16; Both space.size and space.committed_size would be 64, while space.size_in_header would be 65 for the copy of the data directory. The space.size is based on the file size, and we’d never extend files in increments smaller than 4096 bytes. This patch has not gone through our stress tests yet. In any case, only cmake -DCMAKE_BUILD_TYPE=Debug should be affected by this.

            People

              marko Marko Mäkelä
              karolyi László Károlyi
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.