Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25111

Long semaphore wait (> 800 secs), server stops responding

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.3.28
    • N/A

    Description

      We had been running MariaDB 10.3.27 with Galera cluster in production without any issues.

      Less than 26 hours after updating to 10.3.28 one of three servers stopped responding.

      The following errors were logged for more than 800 seconds until I restarted MariaDB:

      2021-03-10 15:10:45 0 [Warning] InnoDB: A long semaphore wait:
      --Thread 140231647176448 has waited at lock0lock.cc line 3882 for 241.00 seconds the semaphore:
      Mutex at 0x5562730bb380, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2
      2021-03-10 15:10:45 0 [Warning] InnoDB: A long semaphore wait:
      --Thread 140286587229952 has waited at lock0lock.cc line 3882 for 241.00 seconds the semaphore:
      Mutex at 0x5562730bb380, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2
      [clip]
      2021-03-10 15:10:45 0 [Note] InnoDB: A semaphore wait:
      --Thread 140232309577472 has waited at row0row.cc line 1133 for 239.00 seconds the semaphore:
      X-lock on RW-latch at 0x7f8f0086a270 created in file buf0buf.cc line 1563
      a writer (thread id 140232091498240) has reserved it in mode  exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time write locked in file row0row.cc line 1133
      [clip]
      2021-03-10 15:10:45 0 [Note] InnoDB: A semaphore wait:
      --Thread 140231088736000 has waited at lock0lock.cc line 3882 for 238.00 seconds the semaphore:
      Mutex at 0x5562730bb380, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2
      2021-03-10 15:10:45 0 [Note] InnoDB: A semaphore wait:
      --Thread 140232326362880 has waited at srv0srv.cc line 2026 for 238.00 seconds the semaphore:
      X-lock (wait_ex) on RW-latch at 0x55627310f3e0 created in file dict0dict.cc line 920
      a writer (thread id 140232326362880) has reserved it in mode  wait exclusive
      number of readers 4, waiters flag 1, lock_word: fffffffc
      Last time write locked in file srv0srv.cc line 2026
      2021-03-10 15:10:45 0 [Note] InnoDB: A semaphore wait:
      --Thread 140231088121600 has waited at lock0lock.cc line 3882 for 238.00 seconds the semaphore:
      Mutex at 0x5562730bb380, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2
      2021-03-10 15:10:45 0 [Note] InnoDB: A semaphore wait:
      --Thread 140231087814400 has waited at lock0lock.cc line 3882 for 238.00 seconds the semaphore:
      Mutex at 0x5562730bb380, Mutex LOCK_SYS created lock0lock.cc:461, lock var 2
      

      I'll attach the full log from when the problem started. The log also contains InnoDB Monitor output.

      This sounds similar to MDEV-24375, but seems to be a different issue since MDEV-24275 should have fixed that in 10.3.28 if I got things right.

      Attachments

        1. mariadb-semaphore-wait.log
          4.74 MB
          Ere Maijala
        2. mysql-galera.log
          366 kB
          JDT

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              emaijala Ere Maijala
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.