Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24606

InnoDB: Semaphore wait has lasted > 600 second

Details

    Description

      After upgrading 10.4.15 to 10.4.17 I experience forced crashes every few days.
      This is zabbix database (like MDEV-24504 ).
      Last crash is from debug build.
      I'll attach full log and backtraces

      Attachments

        1. semaphore-bt.txt
          120 kB
          Marcin Gryszkalis
        2. semaphore-error.log
          5.80 MB
          Marcin Gryszkalis

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24378 Crashes on Semaphore wait > 600 seconds

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24504 [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24275 InnoDB persistent stats analyze forces full scan forcing lock crash

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-25111 Long semaphore wait (> 800 secs), server stops responding

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            marcin.gryszkalis Marcin Gryszkalis created issue -
            marcin.gryszkalis Marcin Gryszkalis added a comment -

            Maybe zabbix is just good for load testing, but more people are reporting: https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/416036-mysql-error-fatal-innodb-semaphore-wait-has-lasted-600-seconds

            marcin.gryszkalis Marcin Gryszkalis added a comment - Maybe zabbix is just good for load testing, but more people are reporting: https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/416036-mysql-error-fatal-innodb-semaphore-wait-has-lasted-600-seconds
            alice Alice Sherepa made changes -
            Field Original Value New Value
            alice Alice Sherepa made changes -
            Assignee Marko Mäkelä [ marko ]
            alice Alice Sherepa made changes -
            marcin.gryszkalis Marcin Gryszkalis added a comment -

            I have 3 more coredumps from last few days (the same version, same installation, same config) - in case you need something to compare.

            marcin.gryszkalis Marcin Gryszkalis added a comment - I have 3 more coredumps from last few days (the same version, same installation, same config) - in case you need something to compare.
            elenst Elena Stepanova made changes -
            Fix Version/s 10.4 [ 22408 ]
            marko Marko Mäkelä added a comment -

            marcin.gryszkalis, thank you. The semaphore-bt.txt suggests that this could be a duplicate of MDEV-24275. Can you please test the latest release (10.4.18)?

            marko Marko Mäkelä added a comment - marcin.gryszkalis , thank you. The semaphore-bt.txt suggests that this could be a duplicate of MDEV-24275 . Can you please test the latest release (10.4.18)?
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Labels semaphore need_feedback semaphore
            marcin.gryszkalis Marcin Gryszkalis added a comment -

            Looks promising - no crashes with 10.4.18 (4 days or so), thanks

            marcin.gryszkalis Marcin Gryszkalis added a comment - Looks promising - no crashes with 10.4.18 (4 days or so), thanks
            marko Marko Mäkelä added a comment -

            marcin.gryszkalis, that sounds good. Please let us know if the hangs stay away, so that we may close this ticket.

            marko Marko Mäkelä added a comment - marcin.gryszkalis , that sounds good. Please let us know if the hangs stay away, so that we may close this ticket.
            emaijala Ere Maijala made changes -
            serg Sergei Golubchik made changes -
            Description After upgrading 10.4.15 to 10.4.17 I experience forced crashes every few days.
            This is zabbix database (like https://jira.mariadb.org/browse/MDEV-24504 ).
            Last crash is from debug build.
            I'll attach full log and backtraces
            		After upgrading 10.4.15 to 10.4.17 I experience forced crashes every few days.
            This is zabbix database (like MDEV-24504 ).
            Last crash is from debug build.
            I'll attach full log and backtraces
            serg Sergei Golubchik made changes -
            Fix Version/s 10.5.9 [ 25109 ]
            Fix Version/s 10.4.18 [ 25110 ]
            Fix Version/s 10.3.28 [ 25111 ]
            Fix Version/s 10.2.37 [ 25112 ]
            Fix Version/s 10.4 [ 22408 ]
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Closed [ 6 ]
            vasilisc Vasilisc added a comment -

            Dec 1 20:10:23 sp mariadbd[84090]: --Thread 140072735258368 has waited at row0ins.cc line 2974 for 618.00 seconds the semaphore:
            Dec 1 20:10:23 sp mariadbd[84090]: X-lock on RW-latch at 0x7f666401ac78 created in file buf0buf.cc line 1227
            Dec 1 20:10:23 sp mariadbd[84090]: a writer (thread id 0) has reserved it in mode SX
            Dec 1 20:10:23 sp mariadbd[84090]: number of readers 0, waiters flag 1, lock_word: 10000000
            Dec 1 20:10:23 sp mariadbd[84090]: Last time write locked in file buf0flu.cc line 837
            Dec 1 20:10:23 sp mariadbd[84090]: InnoDB: Pending reads 0, writes 0
            Dec 1 20:10:23 sp mariadbd[84090]: 2021-12-01 20:10:23 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.

            # dpkg -l|grep maria

            ii libdbd-mariadb-perl 1.21-3 amd64 Perl5 database interface to the MariaDB/MySQL databases
            ii libmariadb3:amd64 1:10.5.12-0+deb11u1 amd64 MariaDB database client library
            ii mariadb-client-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database client binaries
            ii mariadb-client-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core client binaries
            ii mariadb-common 1:10.5.12-0+deb11u1 all MariaDB common configuration files
            ii mariadb-server-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database server binaries
            ii mariadb-server-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core server files

            vasilisc Vasilisc added a comment - Dec 1 20:10:23 sp mariadbd [84090] : --Thread 140072735258368 has waited at row0ins.cc line 2974 for 618.00 seconds the semaphore: Dec 1 20:10:23 sp mariadbd [84090] : X-lock on RW-latch at 0x7f666401ac78 created in file buf0buf.cc line 1227 Dec 1 20:10:23 sp mariadbd [84090] : a writer (thread id 0) has reserved it in mode SX Dec 1 20:10:23 sp mariadbd [84090] : number of readers 0, waiters flag 1, lock_word: 10000000 Dec 1 20:10:23 sp mariadbd [84090] : Last time write locked in file buf0flu.cc line 837 Dec 1 20:10:23 sp mariadbd [84090] : InnoDB: Pending reads 0, writes 0 Dec 1 20:10:23 sp mariadbd [84090] : 2021-12-01 20:10:23 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung. # dpkg -l|grep maria ii libdbd-mariadb-perl 1.21-3 amd64 Perl5 database interface to the MariaDB/MySQL databases ii libmariadb3:amd64 1:10.5.12-0+deb11u1 amd64 MariaDB database client library ii mariadb-client-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database client binaries ii mariadb-client-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core client binaries ii mariadb-common 1:10.5.12-0+deb11u1 all MariaDB common configuration files ii mariadb-server-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database server binaries ii mariadb-server-core-10.5 1:10.5.12-0+deb11u1 amd64 MariaDB database core server files
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 118175 ] MariaDB v4 [ 158782 ]

            People

              marko Marko Mäkelä
              marcin.gryszkalis Marcin Gryszkalis
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.