[MDEV-31783] [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded Created: 2023-07-27  Updated: 2023-10-02  Resolved: 2023-10-02

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.6.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: srinivas chandolu Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Microsoft Windows Server 2016 Standard


Attachments: File WIN-2UUPPCUI6OL.err     File WIN-8FNGQMHRFOS.err     Text File call-stacks-all-threads-07-31.txt     Text File call-stacks-all-threads.txt     File mariadbd.dmp     File mariadbd2.dmp     File my.ini    
Issue Links:
Duplicate
duplicates MDEV-29835 Partial server freeze Closed

 Description   

The MariaDB server has been crashed with the following error after server is up for around 6 days.

2023-07-20 16:54:05 0 [Note] InnoDB: Buffer pool(s) load completed at 230720 16:54:05
2023-07-26 19:11:13 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
230726 19:11:13 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.6.12-MariaDB-log source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc
key_buffer_size=33554432
read_buffer_size=8388608
max_used_connections=94
max_threads=65537
thread_count=94
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1073794283 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
server.dll!my_parameter_handler()
ucrtbase.dll!raise()
ucrtbase.dll!abort()
server.dll!json_valid()
server.dll!json_valid()
server.dll!json_valid()
ntdll.dll!RtlGetAce()
ntdll.dll!RtlReleaseSRWLockExclusive()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file at C:\Program Files\Trellix\IPS Manager\MariaDB\data\
Minidump written to C:\Program Files\Trellix\IPS Manager\MariaDB\data\mariadbd.dmp

Eagerly waiting a reply for the reason for this crash.

Also noticed that a similar issue https://jira.mariadb.org/browse/MDEV-29660 has been created. Whether the above crash is related to this one?



 Comments   
Comment by Marko Mäkelä [ 2023-07-27 ]

oleg.smirnov, can you extract the stack traces from mariadbd.dmp and check if any thread matches one of the stacks posted to MDEV-29835? This hang could be a duplicate of that bug.

Comment by Marko Mäkelä [ 2023-07-28 ]

Based on some data, this does not look like MDEV-29835. I do not see any obvious sign of a deadlock.

In my.ini, I see innodb_buffer_pool_size=512M. How big is the database? What type of storage does it reside in? How much is the total available RAM on the system?

Comment by Marko Mäkelä [ 2023-07-28 ]

Actually, the call-stacks-all-threads.txt that oleg.smirnov produced out of my.ini does look like MDEV-29835. The previous attempts at producing stack traces did not find this:

Callstack for Thread 29 (Thread Id: 5596 (0x15dc)):
10     server.dll!btr_cur_pessimistic_insert…
14     server.dll!btr_cur_pessimistic_insert

This reentrant call matches MDEV-29835. We also have Thread 33 that may be executing btr_cur_t::search_leaf() on the same index tree. I cannot say that for sure, because the .dmp file does not contain the necessary information. I can only say that this is very likely MDEV-29835.

I suggest upgrading to MariaDB Server 10.6.14 or later.

Comment by srinivas chandolu [ 2023-07-31 ]

Marko Mäkelä, Thanks for the update. As suggested will upgrade to 10.6.14.

In between a similar configuration (same my.ini and same system configuration) hit a similar issue. Attached the mariadbd2.dmp for reference. Based on the error looks like a similar issue. Request to check the corresponding stack trace and reason for confirmation.

Comment by Marko Mäkelä [ 2023-07-31 ]

In call-stacks-all-threads-07-31.txt, Thread 13 is executing a recursive call to btr_compress(). There are several other threads waiting in btr_cur_t::search_leaf(). This does look like MDEV-29835 as well.

Comment by Marko Mäkelä [ 2023-08-31 ]

schandol, did an upgrade to 10.6.14 or later fix this issue?

Generated at Thu Feb 08 10:26:26 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.