Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40349

MSAN Builds: slow stack trace on crash delays or prevents core file writing

    XMLWordPrintable

Details

    • Not for Release Notes

    Description

      On MemorySanitizer builds (cmake -DWITH_MSAN=ON), handle_fatal_signal (sql/signal_handler.cc) prints a symbolized stack trace via my_print_stacktrace before it reaches the core dump. Each frame is resolved through the addr2line pipe in mysys/my_addr_resolve.c, which is slow on the large MSAN debug info. Measured on an otherwise idle machine (13.1.0 MSAN debug build): 25 seconds between got signal 6 and Writing a core file in the error log. On a loaded machine it takes longer.

      Any watchdog (i.e. test framework, scripted watchdog, systemd) that stops the server within that window prevents the core file from being written, and a stop landing during the dump itself leaves a truncated core which gdb cannot read. Both outcomes show up regularly in our test runs on MSAN builds; for assertion failures the diagnostics are then reduced to the error log text alone.

      Testcase:

      kill -6 $(pidof mariadbd) 
      

      The error log shows a 25 second gap between got signal 6 and Writing a core file on a server with medium load. Started with --skip-stack-trace, the same signal produces Writing a core file immediately and a complete core about 1 second later.

      The AI proposed fix skips the inline addr2line resolution on MSAN builds. The raw backtrace (backtrace_symbols_fd) is still printed and can be resolved offline, and the core dump starts immediately:

       
      diff --git a/mysys/stacktrace.c b/mysys/stacktrace.c 
      index adce4d0f..15d48af3 100644 
      --- a/mysys/stacktrace.c 
      +++ b/mysys/stacktrace.c 
      @@ -215,7 +215,14 @@ void my_print_stacktrace(uchar* stack_bottom, ulong thread_stack, 
         int n = backtrace(addrs, array_elements(addrs)); 
         my_safe_printf_stderr("stack_bottom = %p thread_stack 0x%lx\n", 
                               stack_bottom, thread_stack); 
      -#if HAVE_MY_ADDR_RESOLVE 
      +/* 
      - On MemorySanitizer builds, resolving the frames via addr2line takes tens 
      - of seconds due to the large debug info, keeping the fatal signal handler 
      - busy before it reaches the core dump. Any supervisor that terminates the 
      - server in that window prevents the core from being written. Print the 
      - raw backtrace instead; it can be resolved offline. 
      +*/ 
      +#if HAVE_MY_ADDR_RESOLVE && !__has_feature(memory_sanitizer) 
      if (print_with_addr_resolve(addrs, n)) 
        return; 
       #endif 
      

      The delay was measured on a 13.1.0 MSAN debug build; the affected code is identical in 13.0 (mysys/stacktrace.c, source revision 172609526b93ac278203607274b70e6b6d8e67fd) and 13.1. The proposed fix was build-tested on a 13.0 MSAN debug build and re-verified with the same signal test: the raw backtrace is printed, Writing a core file follows within 1 second, and the resulting core is complete and readable by gdb. --skip-stack-trace is an effective workaround on unpatched builds.

      I propose we patch 12.3+ MSAN builds CS/ES only.

      Attachments

        Issue Links

          Activity

            People

              danblack Daniel Black
              Roel Roel Van de Paar
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0d
                  0d
                  Logged:
                  Time Spent - 55m
                  55m

                  Git Integration

                    Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.