Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24504

[FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Details

    Description

      We've upgraded several servers from the distro provided MariaDB to 10.5.8. Since then all upgraded servers crash every few days with:

      --Thread 140006271878912 has waited at btr0cur.cc line 1480 for 611.00 seconds the semaphore:
      SX-lock on RW-latch at 0x55aba8228a10 created in file dict0dict.cc line 2161
      a writer (thread id 140005582296832) has reserved it in mode SX
      number of readers 1, waiters flag 1, lock_word: fffffff
      Last time write locked in file dict0stats.cc line 1969

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24188 Hang in buf_page_create() after reusing a previously freed page

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24275 InnoDB persistent stats analyze forces full scan forcing lock crash

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24378 Crashes on Semaphore wait > 600 seconds

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24606 InnoDB: Semaphore wait has lasted > 600 second

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            fhc Fredrik Chabot created issue -
            jck Jacek Kuczynski made changes -
            Field Original Value New Value
            Attachment mariadb_10.5.8_error_log.txt [ 55519 ]
            jck Jacek Kuczynski added a comment -

            We have the very same problem.
            Recently upgraded to 10.5.8 on Ubuntu20.
            Server is on AWS. DB is used for Zabbix.
            We have two MariaDB instances running on the same server. One is using ephemeral SSD storage and there is no problem with that. The other one is using EBS (gp3 [upgraded from gp2 at the same time]) and it's crashing with the same error.

            Attached the last entry in the error log. After crash there are no more entries in the log. Both instances (the crashed one) and the new one are trying to write to the same error log file:

            1. lsof error_mariadb_ebs.log
              COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
              mariadbd 1932433 mysql 1w REG 202,1 828870 518224 error_mariadb_ebs.log
              mariadbd 1932433 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log
              mariadbd 2220904 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log

            After upgrade to 10.5.8 when DB crashed it was fully restarted. After second crash I decided to let it go as it is and after the initial crash there are no more crashes of DB (for last 9 days).

            I will additionally check if this problem is happening also on AWS EBS gp2 type storage.

            jck Jacek Kuczynski added a comment - We have the very same problem. Recently upgraded to 10.5.8 on Ubuntu20. Server is on AWS. DB is used for Zabbix. We have two MariaDB instances running on the same server. One is using ephemeral SSD storage and there is no problem with that. The other one is using EBS (gp3 [upgraded from gp2 at the same time] ) and it's crashing with the same error. Attached the last entry in the error log. After crash there are no more entries in the log. Both instances (the crashed one) and the new one are trying to write to the same error log file: lsof error_mariadb_ebs.log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mariadbd 1932433 mysql 1w REG 202,1 828870 518224 error_mariadb_ebs.log mariadbd 1932433 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log mariadbd 2220904 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log After upgrade to 10.5.8 when DB crashed it was fully restarted. After second crash I decided to let it go as it is and after the initial crash there are no more crashes of DB (for last 9 days). I will additionally check if this problem is happening also on AWS EBS gp2 type storage.
            elenst Elena Stepanova made changes -
            Component/s Storage Engine - InnoDB [ 10129 ]
            Fix Version/s 10.5 [ 23123 ]
            Assignee Marko Mäkelä [ marko ]
            alice Alice Sherepa made changes -
            alice Alice Sherepa made changes -
            marko Marko Mäkelä added a comment -

            Without having stack traces of all threads during the hang, it is impossible to analyze this. This could be a duplicate of MDEV-24188 or MDEV-24275. Does the 10.5.9 release work?

            marko Marko Mäkelä added a comment - Without having stack traces of all threads during the hang, it is impossible to analyze this. This could be a duplicate of MDEV-24188 or MDEV-24275 . Does the 10.5.9 release work?
            marko Marko Mäkelä made changes -
            Labels regression need_feedback regression
            fhc Fredrik Chabot added a comment -

            Version 10.5.9 seems to have resolved this issue.

            fhc Fredrik Chabot added a comment - Version 10.5.9 seems to have resolved this issue.
            elenst Elena Stepanova made changes -
            Fix Version/s 10.5.9 [ 25109 ]
            Fix Version/s 10.4.18 [ 25110 ]
            Fix Version/s 10.3.28 [ 25111 ]
            Fix Version/s 10.2.37 [ 25112 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]
            elenst Elena Stepanova made changes -
            elenst Elena Stepanova made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 117749 ] MariaDB v4 [ 158740 ]

            People

              marko Marko Mäkelä
              fhc Fredrik Chabot
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.