Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14776

InnoDB Monitor output generated by specific error is flooding error logs

    XMLWordPrintable

Details

    Description

      We've ran into an unusual case where InnoDB Monitor output is flooding the error logs, and eventually leading to disk space issues.

      Apparently these cases where the InnoDB Monitor was turned on was triggered by this kind of error message:

      2017-12-19 23:38:26 7f5f729fc700 InnoDB: Warning: difficult to find free blocks in
      		 
      InnoDB: the buffer pool (338 search iterations)!
      		 
      InnoDB: 0 failed attempts to flush a page! Consider
      		 
      InnoDB: increasing the buffer pool size.
      		 
      InnoDB: It is also possible that in your Unix version
      		 
      InnoDB: fsync is very slow, or completely frozen inside
      		 
      InnoDB: the OS kernel. Then upgrading to a newer version
      		 
      InnoDB: of your operating system may help. Look at the
      		 
      InnoDB: number of fsyncs in diagnostic info below.
      		 
      InnoDB: Pending flushes (fsync) log: 0; buffer pool: 0
      		 
      InnoDB: 105756 OS file reads, 62064 OS file writes, 16768 OS fsyncs
      		 
      InnoDB: Starting InnoDB Monitor to print further
      		 
      InnoDB: diagnostics to the standard output.

      Once this occurs, innodb monitor seems to get switched on permanently here, and there does not seem to be a way to stop it that point.

      A restart seems to "fix" it temporarily.

      Also, explicitly setting innodb_status_output to OFF seems to prevent it (after a restart).

      Unlike with long semaphore waits monitor output is enabled permanently here, not just for 30s like with long semaphore waits?

      What exactly should we look for in the generated monitor output to track down the root cause?

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12327 InnoDB status monitor gets enabled after server startup

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12557 INNODB MONITOR OUTPUT write every minutes old messages to error log

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-11215 Several locks taken to same record inside a transaction.

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16339 Upgrading to 10.1.32 shows innodb_empty_free_list_algorithm=BACKOFF as default when it should be 'LEGACY'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16617 After upgrading from 5.5.52 to 10.1.32 getting [Warning] InnoDB: Difficult to find free blocks in the buffer pool

          • Major - Major loss of function.
          • Closed

          Activity

            People

              jplindst Jan Lindström (Inactive)
              ccalender Chris Calender (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.