Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10814

Feature request: Optionally exclude large buffers from core dumps

Details

    Description

      As the size of core dump files by default mostly equals the process size producing core dumps can become an issue on systems with memory buffers, esp. large innodb_buffer_pool_size.

      There needs to be enough file system space to store such large core dumps, and with multi gigabyte files it also takes a non-trivial amount of time to write these, so delaying process restart quite a bit.

      And then there's also a security aspect to it: as the core dump contains the complete innodb buffer pool it contains a substantial amount, or even all, of the actual user data in the database.

      At the same time actual buffer contents are rarely needed when doing a post mortem analysis (usually we only need stack frames and a few pieces of local data).

      So I'm proposing a server option to exclude certain buffers from core dumps by marking them as DONOTDUMP with the madvise() system call.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18946 munmap of 1 byte during shutdown is EINVAL

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-17159 Document what is included in core dumps

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-16605 Always include buf_madvise_do_dump in binaries.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20684 innodb: use madvise CORE/NOCORE on FreeBSD for selective large allocations

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21741 [Warning] InnoDB: Failed to set memory to MADV_DONTDUMP: Function not implemented

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-22186 Please add innodb_buffer_pool_in_core_file option (Facebook/MySQL 8 cherry pick patch)

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 7 older comments
            sanja Oleksandr Byelkin added a comment -

            I checked QC part looks OK, but I do not know how that madwise works, and failed test on github makes me doubting about allowing that changes

            sanja Oleksandr Byelkin added a comment - I checked QC part looks OK, but I do not know how that madwise works, and failed test on github makes me doubting about allowing that changes
            hholzgra Hartmut Holzgraefe added a comment -

            It should not affect mysqld at runtime at all, it is only evaluated when actually writing a core dump:

            From the madvise(2) man page:

            MADV_DONTDUMP (since Linux 3.4)
            Exclude from a core dump those pages in the range specified by
            addr and length. This is useful in applications that have
            large areas of memory that are known not to be useful in a
            core dump. The effect of MADV_DONTDUMP takes precedence over
            the bit mask that is set via the /proc/[pid]/coredump_filter
            file (see core(5)).

            hholzgra Hartmut Holzgraefe added a comment - It should not affect mysqld at runtime at all, it is only evaluated when actually writing a core dump: From the madvise(2) man page: MADV_DONTDUMP (since Linux 3.4) Exclude from a core dump those pages in the range specified by addr and length. This is useful in applications that have large areas of memory that are known not to be useful in a core dump. The effect of MADV_DONTDUMP takes precedence over the bit mask that is set via the /proc/ [pid] /coredump_filter file (see core(5)).
            danblack Daniel Black added a comment -

            Github / Travis-CI tests have been failing for other reasons (MDEV-15838)

            danblack Daniel Black added a comment - Github / Travis-CI tests have been failing for other reasons ( MDEV-15838 )
            marko Marko Mäkelä added a comment -

            kpenza reported in the maria-discuss list that this change may cause error messages to be displayed on startup and shutdown:

            Sep 25 10:40:53 srv1 mysqld: 2018-09-25 10:40:53 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac5400000 size 2097152
            …
            Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432

            The reason for this turned out to be a Linux kernel bug, for which danblack contributed a fix for the Linux 4.19 kernel:

            mm: madvise(MADV_DODUMP): allow hugetlbfs pages

            This was also included in the backport queue for older kernels.

            The messages in the MariaDB server error log can be ignored, and they should disappear after upgrading the kernel.

            marko Marko Mäkelä added a comment - kpenza reported in the maria-discuss list that this change may cause error messages to be displayed on startup and shutdown: Sep 25 10:40:53 srv1 mysqld: 2018-09-25 10:40:53 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac5400000 size 2097152 … Sep 25 10:41:19 srv1 mysqld: 2018-09-25 10:41:19 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aaac3400000 size 33554432 The reason for this turned out to be a Linux kernel bug, for which danblack contributed a fix for the Linux 4.19 kernel : mm: madvise(MADV_DODUMP): allow hugetlbfs pages This was also included in the backport queue for older kernels . The messages in the MariaDB server error log can be ignored, and they should disappear after upgrading the kernel.
            danblack Daniel Black added a comment -

            Notes for anyone else that comes across this:

            The kernel fix has been released in 4.19 and stable kernels 4.18.14, 4.14.76, 4.9.133, 4.4.161, and 3.18.124, and will be in 3.16.62. Redhat has confirmed it will be in their kernels (probably out already).

            Without this fix, the impact is that a core dump may not contain all the information.

            danblack Daniel Black added a comment - Notes for anyone else that comes across this: The kernel fix has been released in 4.19 and stable kernels 4.18.14, 4.14.76, 4.9.133, 4.4.161, and 3.18.124, and will be in 3.16.62. Redhat has confirmed it will be in their kernels (probably out already). Without this fix, the impact is that a core dump may not contain all the information.

            People

              sanja Oleksandr Byelkin
              hholzgra Hartmut Holzgraefe
              Votes:
              4 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.