Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14776

InnoDB Monitor output generated by specific error is flooding error logs

    XMLWordPrintable

Details

    Description

      We've ran into an unusual case where InnoDB Monitor output is flooding the error logs, and eventually leading to disk space issues.

      Apparently these cases where the InnoDB Monitor was turned on was triggered by this kind of error message:

      2017-12-19 23:38:26 7f5f729fc700 InnoDB: Warning: difficult to find free blocks in
      InnoDB: the buffer pool (338 search iterations)!
      InnoDB: 0 failed attempts to flush a page! Consider
      InnoDB: increasing the buffer pool size.
      InnoDB: It is also possible that in your Unix version
      InnoDB: fsync is very slow, or completely frozen inside
      InnoDB: the OS kernel. Then upgrading to a newer version
      InnoDB: of your operating system may help. Look at the
      InnoDB: number of fsyncs in diagnostic info below.
      InnoDB: Pending flushes (fsync) log: 0; buffer pool: 0
      InnoDB: 105756 OS file reads, 62064 OS file writes, 16768 OS fsyncs
      InnoDB: Starting InnoDB Monitor to print further
      InnoDB: diagnostics to the standard output.
      

      Once this occurs, innodb monitor seems to get switched on permanently here, and there does not seem to be a way to stop it that point.

      A restart seems to "fix" it temporarily.

      Also, explicitly setting innodb_status_output to OFF seems to prevent it (after a restart).

      Unlike with long semaphore waits monitor output is enabled permanently here, not just for 30s like with long semaphore waits?

      What exactly should we look for in the generated monitor output to track down the root cause?

      Attachments

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              ccalender Chris Calender (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.