[MDEV-16149]  Failing assertion: node->modification_counter == node->flush_counter with innodb_flush_method=O_DSYNC Created: 2018-05-11  Updated: 2018-05-14  Resolved: 2018-05-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.6
Fix Version/s: 10.3.7

Type: Bug Priority: Major
Reporter: Vladislav Vaintroub Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Problem/Incident
is caused by MDEV-14705 systemd: EXTEND_TIMEOUT_USEC= to avoi... Closed
Relates
relates to MDEV-16150 Mariadb 10.3.6 Failing assertion on D... Closed

 Description   

got this when shutting down server during heavy write benchmark
(innodb_flush_method was set to O_DSYNC)
...
I believe I have seen this assertion also with other innodb_flush_methods in the past (e.g nosync)

-05-11 21:51:59 0x1678  InnoDB: Assertion failure in file D:\winx64-packages\build\src\storage\innobase\fil\fil0fil.cc line 759
InnoDB: Failing assertion: node->modification_counter == node->flush_counter || node->space->purpose == FIL_TYPE_TEMPORARY || srv_fast_shutdown == 2 || !srv_was_started
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/xtradbinnodb-recovery-modes/
InnoDB: about forcing recovery.
180511 21:52:00 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.6-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=64
max_threads=65537
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 212147 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_sigabrt_handler()[my_thr_init.c:485]
mysqld.exe!raise()[signal.cpp:516]
mysqld.exe!abort()[abort.cpp:71]
mysqld.exe!ut_dbg_assertion_failed()[ut0dbg.cc:61]
mysqld.exe!fil_node_close_file()[fil0fil.cc:756]
mysqld.exe!fil_close_all_files()[fil0fil.cc:1945]
mysqld.exe!logs_empty_and_mark_files_at_shutdown()[log0log.cc:2142]
mysqld.exe!innodb_shutdown()[srv0start.cc:2811]
mysqld.exe!innobase_end()[ha_innodb.cc:4269]
mysqld.exe!ha_finalize_handlerton()[handler.cc:463]
mysqld.exe!plugin_deinitialize()[sql_plugin.cc:1258]
mysqld.exe!reap_plugins()[sql_plugin.cc:1334]
mysqld.exe!plugin_shutdown()[sql_plugin.cc:2001]
mysqld.exe!clean_up()[mysqld.cc:2244]
mysqld.exe!unireg_end()[mysqld.cc:2116]
mysqld.exe!kill_server()[mysqld.cc:2042]
mysqld.exe!handle_shutdown()[mysqld.cc:3712]
mysqld.exe!pthread_start()[my_winthread.c:62]
mysqld.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>()[thread.cpp:115]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains



 Comments   
Comment by Elena Stepanova [ 2018-05-13 ]

The version from MDEV-16150 (failure on bootstrap) is reproducible on 10.3.6, but not on the current 10.3. It apparently disappeared after this merge:

commit 99fa7c6c2f9c8d705481bec35f7c4af48848cf70
Merge: 2898c7e b4c5e4a
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date:   Thu Apr 26 22:58:41 2018 +0300
 
    Merge 10.2 into 10.3

but I don't know what exactly in the merge caused it.

Comment by Marko Mäkelä [ 2018-05-14 ]

This was broken in MDEV-14705 and fixed by me when I noticed it while testing MDEV-12218:

10.2 fix (merged to 10.3)

commit b4c5e4a717e3ce2c2d434106cc74417fe9a1d3dc
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date:   Thu Apr 26 22:46:39 2018 +0300
 
    Follow-up fix to MDEV-14705: Flush log at shutdown
    
    In the merge of commit e7f4e61f6e013e8879a64dae3ab0370ec137a121
    the call fil_flush_file_spaces(FIL_TYPE_LOG) is necessary.
    Tablespaces will be flushed as part of the redo log
    checkpoint, but the redo log will not necessarily
    be flushed, depending on innodb_flush_method.

10.1 fix (present in 10.1.33)

commit a1ea8d6f8117179f412799bde1587c210a319b91
Author: Marko Mäkelä <marko.makela@mariadb.com>
Date:   Thu Apr 26 22:44:09 2018 +0300
 
    Follow-up fix to MDEV-14705: Flush log at shutdown
    
    In commit e7f4e61f6e013e8879a64dae3ab0370ec137a121
    the call fil_flush_file_spaces(FIL_LOG) is necessary.
    Tablespaces will be flushed as part of the redo log
    checkpoint, but the redo log will not necessarily
    be flushed, depending on innodb_flush_method.

Only the MariaDB 10.3.6 release is affected. There was no release of MariaDB 10.1 or 10.2 series before I fixed this.

Generated at Thu Feb 08 08:26:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.