Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40117

innodb_fatal_semaphore_wait_threshold crash after thread pool overload

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.4.5
    • None
    • None
    • None
    • Debian 12

    Description

      I am encountering issue, which happens on intermittent basis. Database is crashing with "innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch" error, which is always preceeded by "thread_pool_max_threads" error due to large amount of connection attempts at some point.
      These thread pool errors occur much more often than the semaphore crash, so it is not guaranteed, that the db will always fail, but when does, it is always preceeded by thread pool error.

      Log example:

      2026-06-19 12:24:28 0 [Warning] Threadpool could not create additional thread to handle queries, because the number of allowed threads was reached. Increasing 'thread_pool_max_threads' parameter can help in this situation.
       If 'extra_port' parameter is set, you can still connect to the database with superuser account (it must be TCP connection using extra_port as TCP port) and troubleshoot the situation. A likely cause of pool blocks are clients that lock resources for long time. 'show processlist' or 'show engine innodb status' can give additional hints.
      2026-06-19 12:24:28 0 [Note] Threadpool has been blocked for 30 seconds
       
      2026-06-19 12:30:00 1640815843 [Warning] InnoDB: A long wait (401 seconds) was observed for dict_sys.latch
      2026-06-19 12:33:31 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
      260619 12:33:31 [ERROR] /usr/sbin/mariadbd got signal 6 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report
      a bug on https://jira.mariadb.org/.
       
      Please include the information from the server start above, to the end of the
      information below.
       
      Server version: 11.4.5-MariaDB-deb12-log source revision: 0771110266ff5c04216af4bf1243c65f8c67ccf4
       
      The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
      contains instructions to obtain a better version of the backtrace below.
      Following these instructions will help MariaDB developers provide a fix quicker.
       
      Attempting backtrace. Include this in the bug report.
      (note: Retrieving this information may fail)
       
      Thread pointer: 0x0
      stack_bottom = 0x0 thread_stack 0x49000
      2026-06-19 12:33:41 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55cc7c343c9e]
      /usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55cc7be27739]
      /lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f057a85a050]
      /lib/x86_64-linux-gnu/libc.so.6(+0x8aeec)[0x7f057a8a8eec]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f057a859fb2]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f057a844472]
      /usr/sbin/mariadbd(+0x6ee2b3)[0x55cc7ba212b3]
      /usr/sbin/mariadbd(+0x6e5b00)[0x55cc7ba18b00]
      /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x55cc7c2d93e8]
      /usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x55cc7c2d9cdf]
      /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x4f)[0x55cc7c2d86ef]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f057a6d44a3]
      /lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f057a8a71f5]
      /lib/x86_64-linux-gnu/libc.so.6(+0x1098dc)[0x7f057a9278dc]
      

      There is, ~200GB core file, but due to its size it is not feasible to upload it, unless there is a way how to precook/parse it into somewhat usable form.

      Attachments

        Activity

          People

            shipjain Shipra Jain
            sumark Marek Hlavka
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.