Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14776

InnoDB Monitor output generated by specific error is flooding error logs

Details

    Description

      We've ran into an unusual case where InnoDB Monitor output is flooding the error logs, and eventually leading to disk space issues.

      Apparently these cases where the InnoDB Monitor was turned on was triggered by this kind of error message:

      2017-12-19 23:38:26 7f5f729fc700 InnoDB: Warning: difficult to find free blocks in
      		 
      InnoDB: the buffer pool (338 search iterations)!
      		 
      InnoDB: 0 failed attempts to flush a page! Consider
      		 
      InnoDB: increasing the buffer pool size.
      		 
      InnoDB: It is also possible that in your Unix version
      		 
      InnoDB: fsync is very slow, or completely frozen inside
      		 
      InnoDB: the OS kernel. Then upgrading to a newer version
      		 
      InnoDB: of your operating system may help. Look at the
      		 
      InnoDB: number of fsyncs in diagnostic info below.
      		 
      InnoDB: Pending flushes (fsync) log: 0; buffer pool: 0
      		 
      InnoDB: 105756 OS file reads, 62064 OS file writes, 16768 OS fsyncs
      		 
      InnoDB: Starting InnoDB Monitor to print further
      		 
      InnoDB: diagnostics to the standard output.

      Once this occurs, innodb monitor seems to get switched on permanently here, and there does not seem to be a way to stop it that point.

      A restart seems to "fix" it temporarily.

      Also, explicitly setting innodb_status_output to OFF seems to prevent it (after a restart).

      Unlike with long semaphore waits monitor output is enabled permanently here, not just for 30s like with long semaphore waits?

      What exactly should we look for in the generated monitor output to track down the root cause?

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12327 InnoDB status monitor gets enabled after server startup

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12557 INNODB MONITOR OUTPUT write every minutes old messages to error log

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-11215 Several locks taken to same record inside a transaction.

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16339 Upgrading to 10.1.32 shows innodb_empty_free_list_algorithm=BACKOFF as default when it should be 'LEGACY'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-16617 After upgrading from 5.5.52 to 10.1.32 getting [Warning] InnoDB: Difficult to find free blocks in the buffer pool

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            elenst Elena Stepanova added a comment -

            We already had some complaints about the monitor: MDEV-12557, MDEV-12327. Assigning to jplindst to decide what should be done about this.

            elenst Elena Stepanova added a comment - We already had some complaints about the monitor: MDEV-12557 , MDEV-12327 . Assigning to jplindst to decide what should be done about this.
            jplindst Jan Lindström (Inactive) added a comment -

            https://github.com/MariaDB/server/commit/5ce916ddfcbc9b99f271465d813f595c0e6c5144

            jplindst Jan Lindström (Inactive) added a comment - https://github.com/MariaDB/server/commit/5ce916ddfcbc9b99f271465d813f595c0e6c5144
            jplindst Jan Lindström (Inactive) added a comment -

            commit 07aa98597984391a6e8c85f634106aef4b096a24
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Tue Jan 9 12:37:58 2018 +0200

            MDEV-14776: InnoDB Monitor output generated by specific error is flooding error logs

            innodb/buf_LRU_get_free_block
            Add debug instrumentation to produce error message about
            no free pages. Print error message only once and do not
            enable innodb monitor.

            xtradb/buf_LRU_get_free_block
            Add debug instrumentation to produce error message about
            no free pages. Print error message only once and do not
            enable innodb monitor. Remove code that does not seem to
            be used.

            innodb-lru-force-no-free-page.test
            New test case to force produce desired error message.

            jplindst Jan Lindström (Inactive) added a comment - commit 07aa98597984391a6e8c85f634106aef4b096a24 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Tue Jan 9 12:37:58 2018 +0200 MDEV-14776 : InnoDB Monitor output generated by specific error is flooding error logs innodb/buf_LRU_get_free_block Add debug instrumentation to produce error message about no free pages. Print error message only once and do not enable innodb monitor. xtradb/buf_LRU_get_free_block Add debug instrumentation to produce error message about no free pages. Print error message only once and do not enable innodb monitor. Remove code that does not seem to be used. innodb-lru-force-no-free-page.test New test case to force produce desired error message.

            People

              jplindst Jan Lindström (Inactive)
              ccalender Chris Calender (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.