Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32977

InnoDB Cannot close file Pending operations and pending fsync

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.6.16
    • None
    • None
    • Ubuntu 20, mariadb 10.6.16, fully NVME pcie 4.0, high end server, hundreds databases with hundreds tables

    Description

      Hi,

      we have version 10.6 more than year. Yesterday we updated to 10.6.16 (before 10.6.12), we hoped it will solve our problem, but it doesn't.

      ur mariadb error log is filled repeatelly with problems with pending operations, pending fsync.

      Don't know what's problem. I know, its 'NOTE", not 'WARNING'/'ERROR', but...

      Example of 3 types of "error":
      A) pending fsync:
      [Note] InnoDB: Cannot close file ./XXX/YYY.ibd because of pending fsync

      B) pending operations:
      [Note] InnoDB: Cannot close file ./XXX/YYY.ibd because of 4 pending operations

      C) both:
      [Note] InnoDB: Cannot close file ./XXX/YYY.ibd because of 8 pending operations and pending fsync

      Every time its another database or table, not same. We have about 100 lines per day logged with this.

      We are thinking to upgrade 10.6 to 10.11, but not sure, if this can helps, because 10.6 is still maintained for long time, so...

      If it will necessary, I can provide more informations, ask me if something else is needed.
      MariaDB 10.1 and 10.3 without problem.
      We have high end server and fully nvme pcie 4.0 drivers.
      We have hundreds of databases and every database have hundreds of tables.
      We aren't facing any problems with real performance.
      Ubuntu 20, kernel 5.4.0-156-generic
      2x CPU Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz (Total 40 cores, 80 threads)
      innodb_buffer_pool_size=200G
      innodb_io_capacity=2000
      innodb_io_capacity_max=4000
      Total server RAM is 756G

      Thank you.

      Attachments

        Issue Links

          Activity

            We have a separate support forum for paying customers. I suspect that the log checkpoint pressure due to a small log file size is causing bursts of writes to large numbers of data files. You can just ignore the messages or try to increase the number of open files until the messages become more rare. You may also test how long the typical crash recovery time actually is with some different values of innodb_log_file_size.

            marko Marko Mäkelä added a comment - We have a separate support forum for paying customers. I suspect that the log checkpoint pressure due to a small log file size is causing bursts of writes to large numbers of data files. You can just ignore the messages or try to increase the number of open files until the messages become more rare. You may also test how long the typical crash recovery time actually is with some different values of innodb_log_file_size .
            shaim Dominik Shaim added a comment - - edited

            OK, I understand. Can you share informations about this forum for paying customers?

            What's next, should I close this issue now? (Because I think before, it's any type of bug, but as you already wrote, it's feature)

            Or leave open now and post here update after we will change values? (maybe it will helps someone else as well)

            shaim Dominik Shaim added a comment - - edited OK, I understand. Can you share informations about this forum for paying customers? What's next, should I close this issue now? (Because I think before, it's any type of bug, but as you already wrote, it's feature) Or leave open now and post here update after we will change values? (maybe it will helps someone else as well)
            shaim Dominik Shaim added a comment - - edited

            We changed 3 days ago:
            innodb_open_files 400 -> 512
            innodb_log_file_size 1GB -> 2GB

            But behaviour stayed nearly same, not much changes (tested 3 days, still same).
            Every day approximatelly 60-80 logged records with "pending" text.
            Before approximatelly 80-100, so its not good change, but maybe it's because now is less traffic (Christmas).. so :/

            Any recommendation please?

            Should we try again to increase both of these limits?

            Btw, we updated to mariabd 10.11, still "same" (approximatelly 50 per day) - but now is low traffic because of holiday.

            shaim Dominik Shaim added a comment - - edited We changed 3 days ago: innodb_open_files 400 -> 512 innodb_log_file_size 1GB -> 2GB But behaviour stayed nearly same, not much changes (tested 3 days, still same). Every day approximatelly 60-80 logged records with "pending" text. Before approximatelly 80-100, so its not good change, but maybe it's because now is less traffic (Christmas).. so :/ Any recommendation please? Should we try again to increase both of these limits? Btw, we updated to mariabd 10.11, still "same" (approximatelly 50 per day) - but now is low traffic because of holiday.

            The warning message merely says that the configured innodb_open_files is being exceeded. Closing and reopening files does incur some overhead, and therefore it would make sense to set innodb_open_files as large as possible. The default value makes me think of old things like FILES=30 in the MS-DOS CONFIG.SYS. https://unix.stackexchange.com/questions/84227/limits-on-the-number-of-file-descriptors mentions the Linux /proc/sys/fs/file-max, which identifies a global limit of file descriptors. On my AMD64 system, that is 2⁶³-1, so it does not look like an accurate limit. The first field of the file /proc/sys/fs/file-nr reflects the number of currently open file descriptors, which is 12,802 on my desktop system, with some 1½ hours of uptime and after some light GUI usage.

            I found a definition of the Linux kernel struct file in include/linux/fs.h. I might assume that most of its members are 64 or 128 bits. The member f_path looks a bit suspicious, but it turns out that struct path only encapsulates 2 pointers. There are about 20 data members in struct file in total. It would seem that an open file handle would consume between a couple hundred bytes and a couple kilobytes, depending on the absolute path name lengths, overlay mounts and whatnot. I’m guessing here, but a gigabyte of memory could cover one million open file handles. Based on that, you might considerably increase the innodb_open_files limit.

            Again, this system is for reporting bugs in MariaDB products. Support for paying customers is provided by other means.

            marko Marko Mäkelä added a comment - The warning message merely says that the configured innodb_open_files is being exceeded. Closing and reopening files does incur some overhead, and therefore it would make sense to set innodb_open_files as large as possible. The default value makes me think of old things like FILES=30 in the MS-DOS CONFIG.SYS . https://unix.stackexchange.com/questions/84227/limits-on-the-number-of-file-descriptors mentions the Linux /proc/sys/fs/file-max , which identifies a global limit of file descriptors. On my AMD64 system, that is 2⁶³-1, so it does not look like an accurate limit. The first field of the file /proc/sys/fs/file-nr reflects the number of currently open file descriptors, which is 12,802 on my desktop system, with some 1½ hours of uptime and after some light GUI usage. I found a definition of the Linux kernel struct file in include/linux/fs.h . I might assume that most of its members are 64 or 128 bits. The member f_path looks a bit suspicious, but it turns out that struct path only encapsulates 2 pointers. There are about 20 data members in struct file in total. It would seem that an open file handle would consume between a couple hundred bytes and a couple kilobytes, depending on the absolute path name lengths, overlay mounts and whatnot. I’m guessing here, but a gigabyte of memory could cover one million open file handles. Based on that, you might considerably increase the innodb_open_files limit. Again, this system is for reporting bugs in MariaDB products. Support for paying customers is provided by other means.
            shaim Dominik Shaim added a comment -

            OK, understand, thank you for detailed explanation.

            Should I close this? Or post here update after another increate innodb_open_files?

            shaim Dominik Shaim added a comment - OK, understand, thank you for detailed explanation. Should I close this? Or post here update after another increate innodb_open_files?

            People

              Unassigned Unassigned
              shaim Dominik Shaim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.