Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16149

Failing assertion: node->modification_counter == node->flush_counter with innodb_flush_method=O_DSYNC

Details

    Description

      got this when shutting down server during heavy write benchmark
      (innodb_flush_method was set to O_DSYNC)
      ...
      I believe I have seen this assertion also with other innodb_flush_methods in the past (e.g nosync)

      -05-11 21:51:59 0x1678  InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc line 759
      InnoDB: Failing assertion: node->modification_counter == node->flush_counter || node->space->purpose == FIL_TYPE_TEMPORARY || srv_fast_shutdown == 2 || !srv_was_started
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
      InnoDB: about forcing recovery.
      180511 21:52:00 [ERROR] mysqld got exception 0x80000003 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.3.6-MariaDB
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=64
      max_threads=65537
      thread_count=0
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 212147 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
      mysqld.exe!raise()[signal.cpp:516]
      mysqld.exe!abort()[abort.cpp:71]
      mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:61]
      mysqld.exe!fil_node_close_file()[fil0fil.cc:756]
      mysqld.exe!fil_close_all_files()[fil0fil.cc:1945]
      mysqld.exe!logs_empty_and_mark_files_at_shutdown()[log0log.cc:2142]
      mysqld.exe!innodb_shutdown()[srv0start.cc:2811]
      mysqld.exe!innobase_end()[ha_innodb.cc:4269]
      mysqld.exe!ha_finalize_handlerton()[handler.cc:463]
      mysqld.exe!plugin_deinitialize()[sql_plugin.cc:1258]
      mysqld.exe!reap_plugins()[sql_plugin.cc:1334]
      mysqld.exe!plugin_shutdown()[sql_plugin.cc:2001]
      mysqld.exe!clean_up()[mysqld.cc:2244]
      mysqld.exe!unireg_end()[mysqld.cc:2116]
      mysqld.exe!kill_server()[mysqld.cc:2042]
      mysqld.exe!handle_shutdown()[mysqld.cc:3712]
      mysqld.exe!pthread_start()[my_winthread.c:62]
      mysqld.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>()[thread.cpp:115]
      KERNEL32.DLL!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      

      Attachments

        Issue Links

          Activity

            elenst Elena Stepanova added a comment - - edited

            The version from MDEV-16150 (failure on bootstrap) is reproducible on 10.3.6, but not on the current 10.3. It apparently disappeared after this merge:

            commit 99fa7c6c2f9c8d705481bec35f7c4af48848cf70
            Merge: 2898c7e b4c5e4a
            Author: Marko Mäkelä <marko.makela@mariadb.com>
            Date:   Thu Apr 26 22:58:41 2018 +0300
             
                Merge 10.2 into 10.3
            

            but I don't know what exactly in the merge caused it.

            elenst Elena Stepanova added a comment - - edited The version from MDEV-16150 (failure on bootstrap) is reproducible on 10.3.6, but not on the current 10.3. It apparently disappeared after this merge: commit 99fa7c6c2f9c8d705481bec35f7c4af48848cf70 Merge: 2898c7e b4c5e4a Author: Marko Mäkelä <marko.makela@mariadb.com> Date: Thu Apr 26 22:58:41 2018 +0300   Merge 10.2 into 10.3 but I don't know what exactly in the merge caused it.
            marko Marko Mäkelä added a comment - - edited

            This was broken in MDEV-14705 and fixed by me when I noticed it while testing MDEV-12218:

            10.2 fix (merged to 10.3)

            commit b4c5e4a717e3ce2c2d434106cc74417fe9a1d3dc
            Author: Marko Mäkelä <marko.makela@mariadb.com>
            Date:   Thu Apr 26 22:46:39 2018 +0300
             
                Follow-up fix to MDEV-14705: Flush log at shutdown
                
                In the merge of commit e7f4e61f6e013e8879a64dae3ab0370ec137a121
                the call fil_flush_file_spaces(FIL_TYPE_LOG) is necessary.
                Tablespaces will be flushed as part of the redo log
                checkpoint, but the redo log will not necessarily
                be flushed, depending on innodb_flush_method.
            

            10.1 fix (present in 10.1.33)

            commit a1ea8d6f8117179f412799bde1587c210a319b91
            Author: Marko Mäkelä <marko.makela@mariadb.com>
            Date:   Thu Apr 26 22:44:09 2018 +0300
             
                Follow-up fix to MDEV-14705: Flush log at shutdown
                
                In commit e7f4e61f6e013e8879a64dae3ab0370ec137a121
                the call fil_flush_file_spaces(FIL_LOG) is necessary.
                Tablespaces will be flushed as part of the redo log
                checkpoint, but the redo log will not necessarily
                be flushed, depending on innodb_flush_method.
            

            Only the MariaDB 10.3.6 release is affected. There was no release of MariaDB 10.1 or 10.2 series before I fixed this.

            marko Marko Mäkelä added a comment - - edited This was broken in MDEV-14705 and fixed by me when I noticed it while testing MDEV-12218 : 10.2 fix (merged to 10.3) commit b4c5e4a717e3ce2c2d434106cc74417fe9a1d3dc Author: Marko Mäkelä <marko.makela@mariadb.com> Date: Thu Apr 26 22:46:39 2018 +0300   Follow-up fix to MDEV-14705: Flush log at shutdown In the merge of commit e7f4e61f6e013e8879a64dae3ab0370ec137a121 the call fil_flush_file_spaces(FIL_TYPE_LOG) is necessary. Tablespaces will be flushed as part of the redo log checkpoint, but the redo log will not necessarily be flushed, depending on innodb_flush_method. 10.1 fix (present in 10.1.33) commit a1ea8d6f8117179f412799bde1587c210a319b91 Author: Marko Mäkelä <marko.makela@mariadb.com> Date: Thu Apr 26 22:44:09 2018 +0300   Follow-up fix to MDEV-14705: Flush log at shutdown In commit e7f4e61f6e013e8879a64dae3ab0370ec137a121 the call fil_flush_file_spaces(FIL_LOG) is necessary. Tablespaces will be flushed as part of the redo log checkpoint, but the redo log will not necessarily be flushed, depending on innodb_flush_method. Only the MariaDB 10.3.6 release is affected. There was no release of MariaDB 10.1 or 10.2 series before I fixed this.

            People

              marko Marko Mäkelä
              wlad Vladislav Vaintroub
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.