Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31783

[ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.6.12
    • N/A
    • Server
    • None
    • Microsoft Windows Server 2016 Standard

    Description

      The MariaDB server has been crashed with the following error after server is up for around 6 days.

      2023-07-20 16:54:05 0 [Note] InnoDB: Buffer pool(s) load completed at 230720 16:54:05
      2023-07-26 19:11:13 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
      230726 19:11:13 [ERROR] mysqld got exception 0x80000003 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 10.6.12-MariaDB-log source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc
      key_buffer_size=33554432
      read_buffer_size=8388608
      max_used_connections=94
      max_threads=65537
      thread_count=94
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1073794283 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      server.dll!my_parameter_handler()
      ucrtbase.dll!raise()
      ucrtbase.dll!abort()
      server.dll!json_valid()
      server.dll!json_valid()
      server.dll!json_valid()
      ntdll.dll!RtlGetAce()
      ntdll.dll!RtlReleaseSRWLockExclusive()
      KERNEL32.DLL!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file at C:\Program Files\Trellix\IPS Manager\MariaDB\data\
      Minidump written to C:\Program Files\Trellix\IPS Manager\MariaDB\data\mariadbd.dmp

      Eagerly waiting a reply for the reason for this crash.

      Also noticed that a similar issue https://jira.mariadb.org/browse/MDEV-29660 has been created. Whether the above crash is related to this one?

      Attachments

        1. WIN-8FNGQMHRFOS.err
          35 kB
        2. WIN-2UUPPCUI6OL.err
          41 kB
        3. my.ini
          3 kB
        4. mariadbd2.dmp
          1.29 MB
        5. mariadbd.dmp
          1.31 MB
        6. call-stacks-all-threads-07-31.txt
          191 kB
        7. call-stacks-all-threads.txt
          186 kB

        Issue Links

          Activity

            oleg.smirnov, can you extract the stack traces from mariadbd.dmp and check if any thread matches one of the stacks posted to MDEV-29835? This hang could be a duplicate of that bug.

            marko Marko Mäkelä added a comment - oleg.smirnov , can you extract the stack traces from mariadbd.dmp and check if any thread matches one of the stacks posted to MDEV-29835 ? This hang could be a duplicate of that bug.

            Based on some data, this does not look like MDEV-29835. I do not see any obvious sign of a deadlock.

            In my.ini, I see innodb_buffer_pool_size=512M. How big is the database? What type of storage does it reside in? How much is the total available RAM on the system?

            marko Marko Mäkelä added a comment - Based on some data, this does not look like MDEV-29835 . I do not see any obvious sign of a deadlock. In my.ini , I see innodb_buffer_pool_size=512M . How big is the database? What type of storage does it reside in? How much is the total available RAM on the system?

            Actually, the call-stacks-all-threads.txt that oleg.smirnov produced out of my.ini does look like MDEV-29835. The previous attempts at producing stack traces did not find this:

            Callstack for Thread 29 (Thread Id: 5596 (0x15dc)):
            10     server.dll!btr_cur_pessimistic_insert…
            14     server.dll!btr_cur_pessimistic_insert
            

            This reentrant call matches MDEV-29835. We also have Thread 33 that may be executing btr_cur_t::search_leaf() on the same index tree. I cannot say that for sure, because the .dmp file does not contain the necessary information. I can only say that this is very likely MDEV-29835.

            I suggest upgrading to MariaDB Server 10.6.14 or later.

            marko Marko Mäkelä added a comment - Actually, the call-stacks-all-threads.txt that oleg.smirnov produced out of my.ini does look like MDEV-29835 . The previous attempts at producing stack traces did not find this: Callstack for Thread 29 (Thread Id: 5596 (0x15dc)): … 10 server.dll!btr_cur_pessimistic_insert… … 14 server.dll!btr_cur_pessimistic_insert This reentrant call matches MDEV-29835 . We also have Thread 33 that may be executing btr_cur_t::search_leaf() on the same index tree. I cannot say that for sure, because the .dmp file does not contain the necessary information. I can only say that this is very likely MDEV-29835 . I suggest upgrading to MariaDB Server 10.6.14 or later.
            schandol srinivas chandolu added a comment - - edited

            Marko Mäkelä, Thanks for the update. As suggested will upgrade to 10.6.14.

            In between a similar configuration (same my.ini and same system configuration) hit a similar issue. Attached the mariadbd2.dmp for reference. Based on the error looks like a similar issue. Request to check the corresponding stack trace and reason for confirmation.

            schandol srinivas chandolu added a comment - - edited Marko Mäkelä, Thanks for the update. As suggested will upgrade to 10.6.14. In between a similar configuration (same my.ini and same system configuration) hit a similar issue. Attached the mariadbd2.dmp for reference. Based on the error looks like a similar issue. Request to check the corresponding stack trace and reason for confirmation.

            In call-stacks-all-threads-07-31.txt, Thread 13 is executing a recursive call to btr_compress(). There are several other threads waiting in btr_cur_t::search_leaf(). This does look like MDEV-29835 as well.

            marko Marko Mäkelä added a comment - In call-stacks-all-threads-07-31.txt , Thread 13 is executing a recursive call to btr_compress() . There are several other threads waiting in btr_cur_t::search_leaf() . This does look like MDEV-29835 as well.

            schandol, did an upgrade to 10.6.14 or later fix this issue?

            marko Marko Mäkelä added a comment - schandol , did an upgrade to 10.6.14 or later fix this issue?

            People

              Unassigned Unassigned
              schandol srinivas chandolu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.