[MDEV-15631] [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. Created: 2018-03-22  Updated: 2021-12-23  Resolved: 2021-12-23

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.2.13, 10.3.5
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Sebastian Stamm Assignee: Jan Lindström (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Environment:

Windows Server 2012 R2 , 256GB RAM


Attachments: File my.ini    

 Description   

After mass/bulk insert the DB crashes with the following error:

2018-03-20 17:44:36 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
180320 17:44:36 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.5-MariaDB
key_buffer_size=134217728
read_buffer_size=67108864
max_used_connections=85
max_threads=65537
thread_count=90
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 264089 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_parameter_handler()[my_init.c:260]
mysqld.exe!raise()[signal.cpp:516]
mysqld.exe!abort()[abort.cpp:71]
mysqld.exe!ib::fatal::~fatal()[ut0ut.cc:795]
mysqld.exe!srv_error_monitor_thread()[srv0srv.cc:1864]
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()



 Comments   
Comment by Sebastian Stamm [ 2018-04-02 ]

With 10.2.13 i run into similar problems. With 10.2.14 seems like it's fixed, maybe related to: MDEV-13935 ?

Comment by Marko Mäkelä [ 2018-04-07 ]

sstamm, can you please provide more information, such as stack traces from the server during the 10-minute hang before the InnoDB watchdog finally killed the server? On Windows, procmon should be useful.

One known cause of a watchdog kill is reported in MDEV-15641. Based on the information that has been included so far, we only know that the InnoDB watchdog was tripped, but not the reason why. Was there any preceding output in the error log? Can you share the SHOW CREATE TABLE and some data?

Could this be caused by the InnoDB change buffer? Can you repeat with innodb_change_buffering=none?
wlad has been debugging a hang due to the change buffer on Windows.

Comment by Vladislav Vaintroub [ 2018-04-07 ]

marko meant procdump from https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.

"procdump mysqld.exe" would generate a dump. If you could supply that dump, this would be great. The issue I'm working on is MDEV-15707, which is reincarnation of the old MDEV-77 in 10.2

Comment by Sebastian Stamm [ 2018-04-10 ]

Unfortunately I'm not able to reproduce it at short (switched to 10.2.14).
But by switching from 10.3.5/10.2.13 to 10.2.14 this doesn't happened again. So looks like solved.

Comment by Marko Mäkelä [ 2018-04-15 ]

sstamm, could this be a duplicate of MDEV-13935? I fixed it in 10.2.14, and the fix will be part of 10.3.6 as well.

Generated at Thu Feb 08 08:22:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.