Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26275

Mariadb 10.4.17 crashes with Semaphore > 600 seconds

    XMLWordPrintable

Details

    Description

      Hi Team,

      We have a MariaDB 10.4.17 production slave instance crashing couple of times every week with semaphore wait has lasted > 600 seconds. The master of this instance is running on MySQL 5.6.39-1 version, which doesn't have any issues and is up from the past 154 days. Below are the details
      Below MariaDB version running on Debian 10
      10.4.17-MariaDB-1:10.4.17+maria~stretch-log

      Error message from error log

      2021-07-24 19:29:53 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.
      210724 19:29:53 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 10.4.17-MariaDB-1:10.4.17+maria~stretch-log
      key_buffer_size=16777216
      read_buffer_size=2097152
      max_used_connections=5
      max_threads=402
      thread_count=15
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2496187 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x49000
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5606b742bb5e]
      /usr/sbin/mysqld(handle_fatal_signal+0x3af)[0x5606b6eb9c4f]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fe9b000a730]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fe9af4c87bb]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fe9af4b3535]
      /usr/sbin/mysqld(+0xb7e021)[0x5606b71b1021]
      /usr/sbin/mysqld(+0xb331f2)[0x5606b71661f2]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7fe9afffffa3]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe9af58a4cf]
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /var/lib/mysql
      Resource Limits:
      Fatal signal 11 while backtracing

      Innodb Monitor output related to semaphores

      =====================================
      2021-07-24 19:29:46 0x7fe8a9ad0700 INNODB MONITOR OUTPUT
      =====================================
      Per second averages calculated from the last 7 seconds
      -----------------
      BACKGROUND THREAD
      -----------------
      srv_master_thread loops: 161013 srv_active, 0 srv_shutdown, 9848 srv_idle
      srv_master_thread log flush and writes: 170835
      ----------
      SEMAPHORES
      ----------
      OS WAIT ARRAY INFO: reservation count 3670910
      --Thread 140637230651136 has waited at srv0srv.cc line 2017 for 944.00 seconds the semaphore:
      X-lock (wait_ex) on RW-latch at 0x5606b7de81b0 created in file dict0dict.cc line 833
      a writer (thread id 140637230651136) has reserved it in mode wait exclusive
      number of readers 4, waiters flag 0, lock_word: fffffffc
      Last time write locked in file srv0srv.cc line 2017
      --Thread 140636944971520 has waited at btr0cur.cc line 1492 for 946.00 seconds the semaphore:
      SX-lock on RW-latch at 0x5606be9a10c8 created in file dict0dict.cc line 1954
      a writer (thread id 140637247436544) has reserved it in mode SX
      number of readers 0, waiters flag 1, lock_word: 10000000
      Last time write locked in file dict0stats.cc line 1968
      --Thread 140637213558528 has waited at btr0cur.cc line 1492 for 946.00 seconds the semaphore:
      SX-lock on RW-latch at 0x5606be9a10c8 created in file dict0dict.cc line 1954
      a writer (thread id 140637247436544) has reserved it in mode SX
      number of readers 0, waiters flag 1, lock_word: 10000000
      Last time write locked in file dict0stats.cc line 1968
      --Thread 140637039556352 has waited at btr0cur.cc line 1492 for 946.00 seconds the semaphore:
      SX-lock on RW-latch at 0x5606be9a10c8 created in file dict0dict.cc line 1954
      a writer (thread id 140637247436544) has reserved it in mode SX
      number of readers 0, waiters flag 1, lock_word: 10000000
      Last time write locked in file dict0stats.cc line 1968
      --Thread 140637222258432 has waited at btr0cur.cc line 1492 for 946.00 seconds the semaphore:
      SX-lock on RW-latch at 0x5606be9a10c8 created in file dict0dict.cc line 1954
      a writer (thread id 140637247436544) has reserved it in mode SX
      number of readers 0, waiters flag 1, lock_word: 10000000
      Last time write locked in file dict0stats.cc line 1968
      --Thread 140637047949056 has waited at btr0cur.cc line 1492 for 946.00 seconds the semaphore:
      SX-lock on RW-latch at 0x5606be9a10c8 created in file dict0dict.cc line 1954
      a writer (thread id 140637247436544) has reserved it in mode SX
      number of readers 0, waiters flag 1, lock_word: 10000000
      Last time write locked in file dict0stats.cc line 1968
      OS WAIT ARRAY INFO: signal count 3473703
      RW-shared spins 2504671, rounds 42293906, OS waits 922559
      RW-excl spins 1664250, rounds 12730384, OS waits 239318
      RW-sx spins 490905, rounds 5036270, OS waits 124407
      Spin rounds per wait: 16.89 RW-shared, 7.65 RW-excl, 10.26 RW-sx

      Attachments

        Activity

          People

            marko Marko Mäkelä
            Jaya Karthik Jaya Karthik Karri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.