[MDEV-20738] my_addr_resolve passes invalid offsets to addr2line - Jira

XML

Word

Printable

Details

Type: Bug
Status: Open (View Workflow)
Priority: Major
Resolution: Unresolved
Affects Version/s: 10.2.27, 10.3.18, 10.4.8
Fix Version/s: 10.4
Component/s: Debug, Packaging
Labels:
None
Environment:
RHEL 7

Description

my_addr_resolve is used to print a nice stack trace in the error log when the server crashes.

There are multiple implementations of my_addr_resolve, but the one that is used on a lot of systems seems to be the one that explicitly calls addr2line:

https://github.com/MariaDB/server/blob/mariadb-10.2.27/mysys/my_addr_resolve.c#L140

addr2line can obviously only resolve source code file names and line numbers if debugging information is available:

addr2line translates addresses into file names and line numbers. Given an address in an executable or an offset in a section of a relocatable object, it uses the debugging information to figure out which file name and line number are associated with it.

https://linux.die.net/man/1/addr2line

For our release builds, we do provide debuginfo packages on many systems.

https://mariadb.com/kb/en/library/how-to-produce-a-full-stack-trace-for-mysqld/#installing-debug-info-packages-on-linux

So I would expect this addr2line-based my_addr_resolve to function properly when debuginfo packages are installed. However, it only seems to result in a bunch of Dwarf Error messages. For example:

191003 13:47:40 [ERROR] mysqld got signal 11 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.2.27-MariaDB-log

key_buffer_size=134217728

read_buffer_size=131072

max_used_connections=1

max_threads=153

thread_count=8

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467243 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x0 thread_stack 0x49000

/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5629d9db7ebe]

/usr/sbin/mysqld(handle_fatal_signal+0x30d)[0x5629d983a84d]

addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (2433779713) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (1107141) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (16785643) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1107127) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (16785642) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1107120) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (33562859) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (177878) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Offset (294313484) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Offset (1749572) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (3575603036) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1763467) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (2324119) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1098645947) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (528916376) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (2974745031) greater than or equal to .debug_str size (121235).

sigaction.c:0(__restore_rt)[0x7f40c1525630]

:0(__poll_nocancel)[0x7f40bf8b4c2d]

/usr/sbin/mysqld(_Z26handle_connections_socketsv+0x23f)[0x5629d960e3ff]

/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x295e)[0x5629d961141e]

/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f40bf7e3545]

/usr/sbin/mysqld(+0x44e7bd)[0x5629d96047bd]

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /var/lib/mysql

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            8388608              unlimited            bytes

Max core file size        unlimited            unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             31095                31095                processes

Max open files            65536                65536                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       31095                31095                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: core

addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (2433779713) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (1107141) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (16785643) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1107127) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (16785642) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1107120) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (33562859) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (6514020) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (6516329) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (618123030) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (5390925) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (177878) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Offset (294313484) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (788601601) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Could not find abbrev number 5979.

addr2line: Dwarf Error: Offset (1749572) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (3575603036) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1763467) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (2324119) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1098645947) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (1063717) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (528916376) greater than or equal to .debug_str size (2508136).

addr2line: Dwarf Error: Offset (7173491) greater than or equal to .debug_str size (121235).

addr2line: Dwarf Error: Offset (2974745031) greater than or equal to .debug_str size (121235).

Reproducing the Problem

This problem can be easily reproduced on RHEL 7.

First make sure that the server package and the debuginfo package are installed:

$ sudo rpm -qa | grep -i mariadb

MariaDB-connect-engine-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-test-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-common-10.2.27-1.el7.centos.x86_64

MariaDB-server-10.2.27-1.el7.centos.x86_64

MariaDB-shared-10.2.27-1.el7.centos.x86_64

MariaDB-server-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-backup-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-common-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-compat-10.2.27-1.el7.centos.x86_64

MariaDB-connect-engine-10.2.27-1.el7.centos.x86_64

MariaDB-test-10.2.27-1.el7.centos.x86_64

MariaDB-shared-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-aws-key-management-10.2.20-1.el7.centos.x86_64

MariaDB-client-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-client-10.2.27-1.el7.centos.x86_64

MariaDB-backup-10.2.27-1.el7.centos.x86_64

And start the server:

sudo systemctl start mariadb

And then kill the server:

sudo kill -s 11 $(pidof mysqld)

And then check the error log to see how the addresses in the stack traces were resolved.

Working Around the Problem

Stack traces can still be obtained from a core dump with gdb. Therefore, this problem can be worked around.

First, make sure that the server package and the debuginfo package are installed:

$ sudo rpm -qa | grep -i mariadb

MariaDB-connect-engine-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-test-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-common-10.2.27-1.el7.centos.x86_64

MariaDB-server-10.2.27-1.el7.centos.x86_64

MariaDB-shared-10.2.27-1.el7.centos.x86_64

MariaDB-server-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-backup-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-common-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-compat-10.2.27-1.el7.centos.x86_64

MariaDB-connect-engine-10.2.27-1.el7.centos.x86_64

MariaDB-test-10.2.27-1.el7.centos.x86_64

MariaDB-shared-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-aws-key-management-10.2.20-1.el7.centos.x86_64

MariaDB-client-debuginfo-10.2.27-1.el7.centos.x86_64

MariaDB-client-10.2.27-1.el7.centos.x86_64

MariaDB-backup-10.2.27-1.el7.centos.x86_64

Then, enable core dumps:

https://mariadb.com/kb/en/library/enabling-core-dumps/

Then, when the server crashes again, extract stack traces from the core dump:

sudo gdb --batch --eval-command="thread apply all bt full" /usr/sbin/mysqld /var/lib/mysql/core.932  > mysqld_full_bt_all_threads.txt

https://mariadb.com/kb/en/library/how-to-produce-a-full-stack-trace-for-mysqld/#getting-full-backtraces-for-all-threads-from-a-core-file

Attachments

Issue Links

is caused by

MDEV-6479 stack traces in 10.1

Closed

relates to

MDEV-15781 mariabackup does not produce a nice stacktrace on Linux

Open

MDEV-18893 debuginfo packages missing from Cento{6|7}/RHEL{6|7}/sles12

Closed

MDEV-19707 debuginfo RPM packages do not depend on the RPM packages containing the binaries

Closed

MDEV-29568 libelf (specificly libdw) based stack resolver

Open

MDEV-16194 Stack trace is not resolved on fulltest2

Open

(1 relates to)

Activity

People

Assignee:: Vicențiu Ciorbaru

Reporter:: Geoff Montee (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2019-10-03 18:06

Updated:: 2023-04-27 14:25

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.