[MDEV-33272] innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch Created: 2024-01-18  Updated: 2024-01-22

Status: Needs Feedback
Project: MariaDB Server
Component/s: Server, Storage Engine - InnoDB
Affects Version/s: 10.11.5
Fix Version/s: None

Type: Bug Priority: Major
Reporter: ASHWANI SINGH Assignee: Marko Mäkelä
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Linux


Attachments: File 10.11.5-my.cnf     Text File 10.11.5-mysqld.log     Text File coredump.txt    
Issue Links:
Duplicate
is duplicated by MDEV-33273 Can't start addr2line Closed

 Description   

Hi Team,

We are facing issue with MariaDB Server 10.11.5 which is getting crashed automatically with error Couldn't start addr2line.

Please find attached mysqld.log & my.cnf



 Comments   
Comment by Daniel Black [ 2024-01-18 ]

Hi,

Thanks for the bug report. Failing to start addr2line isn't as big a problem (it just may not be installed). You've got something in the previous message "InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.".

To help us try to work out how this occurred to you can you:

dnf install MariaDB-server-debuginfo
coredumpctl debug
# on the gdb prompt enter the following
thread apply all bt -frame-arguments all full

Then include its output as an attachment here.

Comment by ASHWANI SINGH [ 2024-01-18 ]

Hi Daniel,

We tried to run thread apply all bt -frame-arguments all full inside gdb prompt but getting error mentioned below:
(gdb) thread apply all bt -frame-arguments all full

Thread 71 (LWP 151687):
No symbol "frame" in current context.
(gdb)

But when we executed thread apply all bt we got some output which am attaching as attachment.
coredump.txt

Comment by Daniel Black [ 2024-01-22 ]

Thanks for the information and getting past the frame-arguments difference. Its hard to remember sometimes which OS has the gdb late enough to support this. set print frame-arguments all was the version compatible with older versions.

Seems the 2G limit applied somewhere (possibly systemd-coredump settings) to the coredump has missed the pages most related to the assertion. Apparently ~20G was expected. This obviously is excluding your 300G innodb buffer pool (mariadb excludes this), part of which might be required to resolve this.

If you do notice this before it asserts ([https://mariadb.com/kb/en/innodb-system-variables/#innodb_fatal_semaphore_wait_threshold|innodb_fatal_semaphore_wait_threshold] defaults to 10 minutes), and you have space gcore (from gdb package) can capture the full process memory.

If you have the ability to find and raise this limit before the next occurrence this would be appreciated. Advice on where this is set will help us update our documentation.

Generated at Thu Feb 08 10:37:40 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.