[MDEV-20738] my_addr_resolve passes invalid offsets to addr2line Created: 2019-10-03  Updated: 2023-04-27

Status: Open
Project: MariaDB Server
Component/s: Debug, Packaging
Affects Version/s: 10.2.27, 10.3.18, 10.4.8
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Geoff Montee (Inactive) Assignee: Vicențiu Ciorbaru
Resolution: Unresolved Votes: 0
Labels: None
Environment:

RHEL 7


Issue Links:
Problem/Incident
is caused by MDEV-6479 stack traces in 10.1 Closed
Relates
relates to MDEV-15781 mariabackup does not produce a nice s... Open
relates to MDEV-18893 debuginfo packages missing from Cento... Closed
relates to MDEV-19707 debuginfo RPM packages do not depend ... Closed
relates to MDEV-29568 libelf (specificly libdw) based stack... Open
relates to MDEV-16194 Stack trace is not resolved on fulltest2 Open

 Description   

my_addr_resolve is used to print a nice stack trace in the error log when the server crashes.

There are multiple implementations of my_addr_resolve, but the one that is used on a lot of systems seems to be the one that explicitly calls addr2line:

https://github.com/MariaDB/server/blob/mariadb-10.2.27/mysys/my_addr_resolve.c#L140

addr2line can obviously only resolve source code file names and line numbers if debugging information is available:

addr2line translates addresses into file names and line numbers. Given an address in an executable or an offset in a section of a relocatable object, it uses the debugging information to figure out which file name and line number are associated with it.

https://linux.die.net/man/1/addr2line

For our release builds, we do provide debuginfo packages on many systems.

https://mariadb.com/kb/en/library/how-to-produce-a-full-stack-trace-for-mysqld/#installing-debug-info-packages-on-linux

So I would expect this addr2line-based my_addr_resolve to function properly when debuginfo packages are installed. However, it only seems to result in a bunch of Dwarf Error messages. For example:

191003 13:47:40 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.2.27-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=153
thread_count=8
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5629d9db7ebe]
/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x5629d983a84d]
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (2433779713) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (1107141) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (16785643) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1107127) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (16785642) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1107120) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (33562859) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (177878) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Offset (294313484) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Offset (1749572) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (3575603036) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1763467) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (2324119) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1098645947) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (528916376) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (2974745031) greater than or equal to .debug_str size (121235).
sigaction.c:0(__restore_rt)[0x7f40c1525630]
:0(__poll_nocancel)[0x7f40bf8b4c2d]
/usr/sbin/mysqld(_Z26handle_connections_socketsv+0x23f)[0x5629d960e3ff]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x295e)[0x5629d961141e]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f40bf7e3545]
/usr/sbin/mysqld(+0x44e7bd)[0x5629d96047bd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             31095                31095                processes
Max open files            65536                65536                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       31095                31095                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: core
 
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (2433779713) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (1107141) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (16785643) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1107127) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (16785642) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1107120) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (33562859) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (177878) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Offset (294313484) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Could not find abbrev number 5979.
addr2line: Dwarf Error: Offset (1749572) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (3575603036) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1763467) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (2324119) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1098645947) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (528916376) greater than or equal to .debug_str size (2508136).
addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (121235).
addr2line: Dwarf Error: Offset (2974745031) greater than or equal to .debug_str size (121235).

Reproducing the Problem

This problem can be easily reproduced on RHEL 7.

First make sure that the server package and the debuginfo package are installed:

$ sudo rpm -qa | grep -i mariadb
MariaDB-connect-engine-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-test-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-common-10.2.27-1.el7.centos.x86_64
MariaDB-server-10.2.27-1.el7.centos.x86_64
MariaDB-shared-10.2.27-1.el7.centos.x86_64
MariaDB-server-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-backup-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-common-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-compat-10.2.27-1.el7.centos.x86_64
MariaDB-connect-engine-10.2.27-1.el7.centos.x86_64
MariaDB-test-10.2.27-1.el7.centos.x86_64
MariaDB-shared-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-aws-key-management-10.2.20-1.el7.centos.x86_64
MariaDB-client-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-client-10.2.27-1.el7.centos.x86_64
MariaDB-backup-10.2.27-1.el7.centos.x86_64

And start the server:

sudo systemctl start mariadb

And then kill the server:

sudo kill -s 11 $(pidof mysqld)

And then check the error log to see how the addresses in the stack traces were resolved.

Working Around the Problem

Stack traces can still be obtained from a core dump with gdb. Therefore, this problem can be worked around.

First, make sure that the server package and the debuginfo package are installed:

$ sudo rpm -qa | grep -i mariadb
MariaDB-connect-engine-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-test-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-common-10.2.27-1.el7.centos.x86_64
MariaDB-server-10.2.27-1.el7.centos.x86_64
MariaDB-shared-10.2.27-1.el7.centos.x86_64
MariaDB-server-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-backup-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-common-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-compat-10.2.27-1.el7.centos.x86_64
MariaDB-connect-engine-10.2.27-1.el7.centos.x86_64
MariaDB-test-10.2.27-1.el7.centos.x86_64
MariaDB-shared-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-aws-key-management-10.2.20-1.el7.centos.x86_64
MariaDB-client-debuginfo-10.2.27-1.el7.centos.x86_64
MariaDB-client-10.2.27-1.el7.centos.x86_64
MariaDB-backup-10.2.27-1.el7.centos.x86_64

Then, enable core dumps:

https://mariadb.com/kb/en/library/enabling-core-dumps/

Then, when the server crashes again, extract stack traces from the core dump:

sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld /var/lib/mysql/core.932  > mysqld_full_bt_all_threads.txt

https://mariadb.com/kb/en/library/how-to-produce-a-full-stack-trace-for-mysqld/#getting-full-backtraces-for-all-threads-from-a-core-file



 Comments   
Comment by Vicențiu Ciorbaru [ 2020-10-23 ]

This is not a problem with regards to addr2line or my_addr_resolve, rather something is not correct between debuginfo packages and the binary package.

Doing an objdump -S /usr/sbin/mysqld with debuginfo packages installed (10.2.34 as of this comment) will lead to the same warning messages of objdump: Dwarf Error: Offset (3987472846) greater than or equal to .debug_str size (2535356).

Comment by Vladislav Lesin [ 2023-03-14 ]

This can also relate to wrong debug info search paths bug: https://bugs.launchpad.net/ubuntu/+source/binutils/+bug/1759248. The workaround is to build packages with embedded debug info instead of stripping binaries and building separate debug info packages. It can be done with:

SET(CMAKE_BUILD_TYPE RelWithDebInfo)
OPTION(CPACK_RPM_DEBUGINFO_PACKAGE "" OFF)

code in the corresponding cmake/build_configurations/*.cmake config files.

UPDATE: I have just read the description more carefully and understood that it does not relate to wrong debug info search path issue. But the comment can still be useful.

Comment by Vladislav Lesin [ 2023-03-14 ]

As an alternative of addr2line we could use libbacktrace(https://github.com/ianlancetaylor/libbacktrace), libunwind(https://github.com/libunwind/libunwind) or Boost.Stacktrace.

Comment by Daniel Black [ 2023-03-14 ]

or libelf (from libdw deb package) (MDEV-29568)

Comment by Anel Husakovic [ 2023-03-24 ]

It seems binutils fix has been applied to Ubuntu 22.04. with binutils 2.38 but not backported.

Generated at Thu Feb 08 09:01:47 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.