Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24504

[FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

Details

    Description

      We've upgraded several servers from the distro provided MariaDB to 10.5.8. Since then all upgraded servers crash every few days with:

      --Thread 140006271878912 has waited at btr0cur.cc line 1480 for 611.00 seconds the semaphore:
      SX-lock on RW-latch at 0x55aba8228a10 created in file dict0dict.cc line 2161
      a writer (thread id 140005582296832) has reserved it in mode SX
      number of readers 1, waiters flag 1, lock_word: fffffff
      Last time write locked in file dict0stats.cc line 1969

      Attachments

        Issue Links

          Activity

            We have the very same problem.
            Recently upgraded to 10.5.8 on Ubuntu20.
            Server is on AWS. DB is used for Zabbix.
            We have two MariaDB instances running on the same server. One is using ephemeral SSD storage and there is no problem with that. The other one is using EBS (gp3 [upgraded from gp2 at the same time]) and it's crashing with the same error.

            Attached the last entry in the error log. After crash there are no more entries in the log. Both instances (the crashed one) and the new one are trying to write to the same error log file:

            1. lsof error_mariadb_ebs.log
              COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
              mariadbd 1932433 mysql 1w REG 202,1 828870 518224 error_mariadb_ebs.log
              mariadbd 1932433 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log
              mariadbd 2220904 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log

            After upgrade to 10.5.8 when DB crashed it was fully restarted. After second crash I decided to let it go as it is and after the initial crash there are no more crashes of DB (for last 9 days).

            I will additionally check if this problem is happening also on AWS EBS gp2 type storage.

            jck Jacek Kuczynski added a comment - We have the very same problem. Recently upgraded to 10.5.8 on Ubuntu20. Server is on AWS. DB is used for Zabbix. We have two MariaDB instances running on the same server. One is using ephemeral SSD storage and there is no problem with that. The other one is using EBS (gp3 [upgraded from gp2 at the same time] ) and it's crashing with the same error. Attached the last entry in the error log. After crash there are no more entries in the log. Both instances (the crashed one) and the new one are trying to write to the same error log file: lsof error_mariadb_ebs.log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME mariadbd 1932433 mysql 1w REG 202,1 828870 518224 error_mariadb_ebs.log mariadbd 1932433 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log mariadbd 2220904 mysql 2w REG 202,1 828870 518224 error_mariadb_ebs.log After upgrade to 10.5.8 when DB crashed it was fully restarted. After second crash I decided to let it go as it is and after the initial crash there are no more crashes of DB (for last 9 days). I will additionally check if this problem is happening also on AWS EBS gp2 type storage.

            Without having stack traces of all threads during the hang, it is impossible to analyze this. This could be a duplicate of MDEV-24188 or MDEV-24275. Does the 10.5.9 release work?

            marko Marko Mäkelä added a comment - Without having stack traces of all threads during the hang, it is impossible to analyze this. This could be a duplicate of MDEV-24188 or MDEV-24275 . Does the 10.5.9 release work?

            Version 10.5.9 seems to have resolved this issue.

            fhc Fredrik Chabot added a comment - Version 10.5.9 seems to have resolved this issue.

            People

              marko Marko Mäkelä
              fhc Fredrik Chabot
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.