Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Duplicate
-
12.3, 13.0, 13.1
-
Not for Release Notes
Description
On MemorySanitizer builds (cmake -DWITH_MSAN=ON), handle_fatal_signal (sql/signal_handler.cc) prints a symbolized stack trace via my_print_stacktrace before it reaches the core dump. Each frame is resolved through the addr2line pipe in mysys/my_addr_resolve.c, which is slow on the large MSAN debug info. Measured on an otherwise idle machine (13.1.0 MSAN debug build): 25 seconds between got signal 6 and Writing a core file in the error log. On a loaded machine it takes longer.
Any watchdog (i.e. test framework, scripted watchdog, systemd) that stops the server within that window prevents the core file from being written, and a stop landing during the dump itself leaves a truncated core which gdb cannot read. Both outcomes show up regularly in our test runs on MSAN builds; for assertion failures the diagnostics are then reduced to the error log text alone.
Testcase:
kill -6 $(pidof mariadbd) |
The error log shows a 25 second gap between got signal 6 and Writing a core file on a server with medium load. Started with --skip-stack-trace, the same signal produces Writing a core file immediately and a complete core about 1 second later.
The AI proposed fix skips the inline addr2line resolution on MSAN builds. The raw backtrace (backtrace_symbols_fd) is still printed and can be resolved offline, and the core dump starts immediately:
|
diff --git a/mysys/stacktrace.c b/mysys/stacktrace.c
|
index adce4d0f..15d48af3 100644
|
--- a/mysys/stacktrace.c
|
+++ b/mysys/stacktrace.c
|
@@ -215,7 +215,14 @@ void my_print_stacktrace(uchar* stack_bottom, ulong thread_stack,
|
int n = backtrace(addrs, array_elements(addrs));
|
my_safe_printf_stderr("stack_bottom = %p thread_stack 0x%lx\n",
|
stack_bottom, thread_stack);
|
-#if HAVE_MY_ADDR_RESOLVE
|
+/*
|
- On MemorySanitizer builds, resolving the frames via addr2line takes tens
|
- of seconds due to the large debug info, keeping the fatal signal handler
|
- busy before it reaches the core dump. Any supervisor that terminates the
|
- server in that window prevents the core from being written. Print the
|
- raw backtrace instead; it can be resolved offline.
|
+*/
|
+#if HAVE_MY_ADDR_RESOLVE && !__has_feature(memory_sanitizer)
|
if (print_with_addr_resolve(addrs, n))
|
return;
|
#endif |
The delay was measured on a 13.1.0 MSAN debug build; the affected code is identical in 13.0 (mysys/stacktrace.c, source revision 172609526b93ac278203607274b70e6b6d8e67fd) and 13.1. The proposed fix was build-tested on a 13.0 MSAN debug build and re-verified with the same signal test: the raw backtrace is printed, Writing a core file follows within 1 second, and the resulting core is complete and readable by gdb. --skip-stack-trace is an effective workaround on unpatched builds.
I propose we patch 12.3+ MSAN builds CS/ES only.
Attachments
Issue Links
- duplicates
-
MDEV-39113 addr2line stack trace resolver deterimental in MariaDB-backup and MSAN/ASAN builds
-
- In Review
-
- relates to
-
MDEV-39113 addr2line stack trace resolver deterimental in MariaDB-backup and MSAN/ASAN builds
-
- In Review
-