Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21418

[Warning] InnoDB: A long semaphore wait

Details

    Description

      "[Warning] InnoDB: A long semaphore wait" messages logged in /var/log syslog.

      MariaDB eventually crashes and restarts;

      mysqld[926]: where mysqld died. If you see no messages after this, something went
      Jan 3 04:39:17 localhost mysqld[926]: terribly wrong...

      Attachments

        1. syslog
          3.24 MB
          Dermot Brereton
        2. syslog.1
          6.67 MB
          Dermot Brereton

        Issue Links

          Activity

            Dermot.Brereton Dermot Brereton added a comment - - edited

            MariaDB [(none)]> select @@innodb_adaptive_hash_index;
            +------------------------------+
            | @@innodb_adaptive_hash_index |
            +------------------------------+
            |                            1 |
            +------------------------------+
            

            Dermot.Brereton Dermot Brereton added a comment - - edited MariaDB [(none)]> select @@innodb_adaptive_hash_index; +------------------------------+ | @@innodb_adaptive_hash_index | +------------------------------+ | 1 | +------------------------------+

            There are know problems with innodb_adaptive_hash_index; see MDEV-20487 and related bugs.

            Dermot.Brereton, if you see hangs with innodb_adaptive_hash_index=0, please run

            gdb -ex "set pagination 0" -ex "thread apply all bt" -ex "print dict_operation_lock" -ex "print dict_sys->mutex" --batch -p $(pgrep -x mysqld)
            

            and try to create and preserve a core dump for further analysis.

            marko Marko Mäkelä added a comment - There are know problems with innodb_adaptive_hash_index ; see MDEV-20487 and related bugs. Dermot.Brereton , if you see hangs with innodb_adaptive_hash_index=0 , please run gdb -ex "set pagination 0" -ex "thread apply all bt" -ex "print dict_operation_lock" -ex "print dict_sys->mutex" --batch -p $(pgrep -x mysqld) and try to create and preserve a core dump for further analysis.
            Dermot.Brereton Dermot Brereton added a comment - - edited

            I have no evidence from the attached logs that this issue is related to innodb_adaptive_hash_index

            Do you recommend setting innodb_adaptive_hash_index = 0 (the current value is 1)?

            Also based on the information provided in the attached syslog's is this issue similar to MDEV-13983?

            Dermot.Brereton Dermot Brereton added a comment - - edited I have no evidence from the attached logs that this issue is related to innodb_adaptive_hash_index Do you recommend setting innodb_adaptive_hash_index = 0 (the current value is 1)? Also based on the information provided in the attached syslog's is this issue similar to MDEV-13983 ?

            Dermot.Brereton, it is hard to analyze hangs solely based on server error log output. It could easily take more than 1 hour of my time, and still the cause could remain unresolved. Stack traces of all threads are a much more convenient and reliable way of analyzing hangs. I cannot say for sure which hang reports are related, because I have not seen the requested gdb output in any of them.

            My motivation to disable the adaptive hash index in MariaDB Server 10.5 by default (MDEV-20487) is the bugs that we are aware of, but have not had time to fix yet. In fact, I would have disabled it at compilation time, but we got some user and customer feedback that it can actually help the performance of read-mostly workloads.

            marko Marko Mäkelä added a comment - Dermot.Brereton , it is hard to analyze hangs solely based on server error log output. It could easily take more than 1 hour of my time, and still the cause could remain unresolved. Stack traces of all threads are a much more convenient and reliable way of analyzing hangs. I cannot say for sure which hang reports are related, because I have not seen the requested gdb output in any of them. My motivation to disable the adaptive hash index in MariaDB Server 10.5 by default ( MDEV-20487 ) is the bugs that we are aware of, but have not had time to fix yet. In fact, I would have disabled it at compilation time, but we got some user and customer feedback that it can actually help the performance of read-mostly workloads.

            Your crash could have the same underlying reason as in MDEV-16467 that was fixed in 10.2.25

            Try upgrading

            serg Sergei Golubchik added a comment - Your crash could have the same underlying reason as in MDEV-16467 that was fixed in 10.2.25 Try upgrading

            Issue was resolved when the DRBD replication software was removed from the database server and the data file system created as ext4.

            Dermot.Brereton Dermot Brereton added a comment - Issue was resolved when the DRBD replication software was removed from the database server and the data file system created as ext4.

            This ticket can now be closed.

            Dermot.Brereton Dermot Brereton added a comment - This ticket can now be closed.

            People

              Unassigned Unassigned
              Dermot.Brereton Dermot Brereton
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.